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(54) Titie: STREAMING HYPER VIDEO AND DYNAMIC HYPERVIDEO 
(57) Abstract 

A data stream comprises a hypervideo data stream including 
hypervideo data associated with an instance in time in a video. In another 
embodiment, the hypervideo data stream further comprises target data 
associated with the instance in time. A method of creating a multimedia 
stream comprises authoring a hypervideo, including a video. Hypervideo 
parameters are exported into a hypervideo data stream of a media file. 
In another embodiment, a data stream of the video is exported into the 
media file. In one embodiment, a method for authoring a hypervideo that 
is dynamic, comprises authoring a hypervideo. and defining a hypervideo 
parameter that may vaiy during the hypervideo performance. In another 
embodiment, an event handler is defined that issues a command to vaiy 
the parameter upon receiving an event message from the hypervideo. 
In one embodiment, a method of performing a hypervideo that is 
dynamic, comprises perfomiing a hypervideo, and varying a parameter 
of the hypervideo during the performance of the hypervideo. In another 
embodiment, the parameter is varied upon the occurrence of an event in the 
hypervideo. In one embodiment, a hypervideo system comprises a dynamic 
hypervideo server. A player is coupled to the dynamic hypervideo server. 
A media server is coupled to the player. 
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Streaming Hypervideo and Dynamic Hypervideo 

A portion of the disclosure of this patent document contains material 
which is subject to copyright protection. The copyright owner has no objection 
5 to the facsimile reproduction by anyone of the patent disclosure, as it appears in 
the Patent and Trademark Office patent files or records, but otherwise reserves all 
copyrights whatsoever. 

Fy^ld Qf the Ipventippi 
1 0 The present invention relates generally to electronic technology, and more 

specifically to interactive multimedia technology. 

P^clfgrPWltf ftf the jnveptPPP 
Computers are capable of conmiunicating information to humans in many 
15 formats, including text, graphics, sound and video. A multimedia presentation on 
a computer combines such fomiats to present infomiation more coherently so that 
it is better perceived by humans. 

Information on computers can be linked. For example, using "hypertext," 
the existence of additional and related infomiation that is associated with a 
20 selected portion of text may be indicated by highlighting or underlining the 
selected text. The information associated with such selected text may be 
accessed, for example, utilizing a pointer device, such as a mouse. A mouse 
button may be actuated when a pointer is positioned on the highlighted text. 
After selecting the highlighted text in such a manner, the user is presented with 
25 additional relevant information. 

However, text is not always the most desirable means of conveying 
infomiation to humans. Some information is best illustrated with video. 
However, xmlike text, there has not been an effective means for implementing 
links in a video to access infomiation. 

30 
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Sammary of the ][nv?ntiop 
The present invention solves the above-mentioned problems in the art and 
other problems which wall be understood by those skilled in the art upon reading 
and understanding the present specification. The present invention is a system 
5 and method for streaming hypervideo data and for implementing dynamic 
hypervideos. 

In one embodiment, a data stream comprises a hypervideo data stream 
including hypervideo data associated with an instance in time in a video. In 
another embodiment, the hypervideo data stream further comprises target data 

1 0 associated with the instance in time. In yet another embodiment, the hypervideo 
data stream further comprises geometry data of a hotspot associated with the 
instance in time. In yet a further embodiment, the hypervideo data stream further 
comprises shape data of a hotspot associated vnth the instance of time. In 
another embodiment, the instance in time corresponds to a frame of the video. 

15 In a further embodiment, a multimedia stream comprises a hypervideo 

data stream that includes a header, including static hypervideo data. A body, 
coupled to the header, includes dynamic hypervideo data. The dynamic 
hypervideo data includes time data that is associated with an instance of time in a 
video with which the dynamic hypervideo data and the static hypervideo data are 

20 associated. 

In another embodiment, the static hypervideo data comprises target data. 

In yet a further embodiment, the dynamic hypervideo data further comprises 

geometry data of a hotspot. In another embodiment, the dynamic hypervideo 

data further comprises shape data of a hotspot. In yet a further embodiment, a 
25 video stream includes a video frame, wherein the instance in time corresponds to 

the video frame. In another embodiment, the multimedia stream further 

comprises an audio stream. 

In one embodiment, a method of creating a multimedia stream comprises 

authoring a hypervideo, including a video. Hypervideo parameters are exported 
30 into a hypervideo data stream of a media file. In another embodiment, a data 

stream of the video is exported into the media file. 
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In one embodiment, a method for authoring a hypervideo that is dynamic, 
comprises authoring a hypervideo, and defining a hypervideo parameter that may 
vary during the hypervideo performance. 

In another embodiment, an event handler is defined that issues a command 
5 to vary the parameter upon receiving an event message fi'om the hypervideo. In 
yet another embodiment, an event handler is defined by a logical condition 
including a variable defined by a query to a database. In a fiirther embodiment, 
the hypervideo is authored by defining a hotspot and linking the hotspot to a 
target. 

1 0 In one embodiment, a method of performing a hypervideo that is dynamic, 

comprises performing a hypervideo, and varying a parameter of the hypervideo 
during the performance of the hypervideo. In another embodiment, the parameter 
is varied upon the occurrence of an event in the hypervideo. In a fiirther 
embodiment, the parameter is varied upon an event handler receiving a message 

1 5 associated with the event, and issuing a command to vary the parameter. In yet 
another embodiment, the parameter is varied upon the event handler evaluating a 
logical condition including a variable defined by a query to a database. In yet a 
fiirther embodiment, the parameter identifies a target. 

In one embodiment, a hypervideo system comprises a dynamic hypervideo 

20 server. A player is coupled to the dynamic hypervideo server. A media server is 
coupled to the player. In another embodiment, the hypervideo system fiirther 
comprises a database. In a fiuther embodiment, the hypervideo system further 
comprises an application program. In another embodiment, the application 
program is an electronic commerce server. In a fiirther embodiment, the 

25 application program is an advertising server. In another embodiment, the 
application program is a training server. In a fiirther embodiment, the media 
server is a video server. In another embodiment, the hypervideo system further 
comprises a network coupling the player to the dynamic hypervideo server and 
the media server. 

30 It is a benefit of the present invention that it facilitates streaming 

hypervideo data over networks. It is a fiirther benefit of the present invention 
that it permits the parameters of a hypervideo to be varied during the hypervideo 
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performance. Further features and advantages of the present invention, as well as 
the structure and operation of various embodiments of the present invention, are 
described in detail below with reference to the accompanying drawings. 



5 Brief Description of the Drawings 

Figure 1 A illustrates an exemplary hypervideo environment, including a 
hypervideo authoring tool; 

Figure IB illustrates an exemplary hypervideo run-time module; 

Figure IC illustrates an exemplary block diagram of one embodiment of a 
1 0 computer system; 

Figure ID exemplifies a computer implemented as an integrated circuit; 

Figure IE illustrates an exemplary television; 

Figure 2 illustrates an exemplary Import Media File Dialog; 

Figure 3 illustrates an exemplary Import From Project Dialog; 
15 Figure 4 illustrates an exemplary Media Warehouse Window Icon; 

Figure 5 illustrates an exemplary Details Page; 

Figure 6 illustrates an exemplary Preview Page; 

Figure 7A illustrates an exemplary Targets Page; 

Figure 7B illustrates an exemplary Hotspot Page; 
20 Figure 7C illustrates an exemplary General Properties Page; 

Figure 7D illustrates an exemplary Time Line; 

Figure 7E illustrates another exemplary Details Page; 

Figure 7F illustrates an exemplary Frame Window; 

Figure 7G illustrates an exemplary Place Properties Page; 
25 Figure 8A illustrates an exemplary Workshop Window; 

Figure 8B illustrates an exemplary Preview Window; 

Figure 9A illustrates an exemplary Tools Window; 

Figure 9B illustrates an exemplary Tracking Property Sheet; 

Figure 10 illustrates an exemplary graphical user interface (GUI) for a 
30 hypervideo authoring tool; 

Figure 1 1 illustrates an exemplary hypervideo story board; 

Figure 12 illustrates an exemplary call-back object; 



Figure 13 iUustrates an exemplary object tree; 

Figure 14 illustrates an exemplary hypervideo data stream; 

Figure 1 5 illustrates an exemplary GUI of a modified version of an 
authoring tool for Netshow (TM); 

Figure 16 illustrates an exemplary Hotspot Properties Sheet; 

Figure 17 illustrates an exemplary Media Properties Sheet; 

Figiu-e 18 illustrates an exemplary Properties Page; 

Figure 19 illustrates an exemplary Active X (TM) control; 

Figure 20 illustrates an exemplary GUI of a modified version of an 
authoring tool for RealVideo (TM); 

Figure 21 illustrates an exemplary Hotspot Properties Sheet; 

Figure 22 illustrates an exemplary Export Dialog; 

Figure 23 illustrates an exemplary Export Dialog for automatic 
translation; 

Figure 24A illustrates an exemplary Encoding Properties Sheet; 

Figure 24B illustrates an exemplary Encoding Properties Page; 

Figure 25 illustrates an exemplary hypervideo system; 

Figure 26 illustrates an exemplary authoring tool GUI; 

Figure 27 illustrates an exemplary Data Source page of Query dialog; 

Figure 28 illustrates an exemplary Choose Colunms Page; 

Figure 29 illustrates an exemplary Filter Data Page; 

Figure 30 illustrates an exemplary SQL Page; 

Figure 31 illustrates an exemplary Targets Page; 

Figure 32 illustrates an exemplary Hotframes toolbar; 

Figure 33 illustrates an exemplary Hotframes Property Sheet; 

Figure 34 illustrates an exemplary Properties Sheet; 

Figure 35 illustrates an exemplary Handlers Notebook Window; 

Figure 36 illustrates an exemplary Script Wizard; 

Figure 37 illustrates an exemplary Menu; 

Figure 38 illustrates an exemplary Defauh Settings Sheet; 

Figure 39 illustrates an exemplary Logging Page; 
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Figure 40 illustrates an exemplary flow diagram illustrating the procedure 
for creating a hypervideo; 

Figure 41 illustrates exemplary logical class relationships associated with 
the Media Warehouse Window; 
5 Figure 42 illustrates exemplary classes associated with class EClipDoc; 



1 5 reference is made to the accompanying drawings which form a part hereof, and in 
which is shown by way of illustration specific preferred embodiments in which the 
invention may be practiced. These embodiments are described in sufficient detail 
to enable persons skilled in the art to practice the invention, and it is to be 
understood that other embodiments may be utilized and that logical, mechanical 

20 and electrical changes may be made without departing from the scope of the 
present invention. The following detailed description is, therefore, not to be 
taken in a limiting sense, and the scope of the present invention is defined only by 
the appended claims. 



10 



Figure 43 
Figure 44 
Figure 45 
Figure 46 
Figure 47 
Figure 48 



illustrates exemplary classes associated with class EClipView; 
illustrates exemplary classes associated with EClipFrameWnd. 
illustrates an exemplary run-time tree class diagram; 
illustrates an exemplary video object class diagram; 
illustrates an exemplary sound object class diagram; and 
illustrates exemplary classes associated with a hypervideo. 



Detailed Description of the Embodiments 

In the following detailed description of the preferred embodiments. 
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1.0 Introduction 

The present invention is a method and apparatus for linking information to 
and accessing information from a video. The video may be hypervideo. 
Hypervideo is video with one or more regions of interest, where each region of 
5 interest may be linked to one or more targets. Hypervideo permits a user to 
interact with video. A user can create hypervideo which is non-linear. Asa 
result, for example, the user can navigate the hypervideo from a base target, such 
as, but not limited to, a video, to other target(s), such as, but not limited to, 
HTML files or other videos. The other targets may be executed, or activated, 

1 0 simultaneously, sequentially, or a combination thereof. The parent target can be 
halted, for example when paused or closed, or keep playing. 

The present invention comprises a method and apparatus for creating and 
playing hypervideo. The present invention may be implemented with computer 
programs. The program that creates hypervideo is known as an authoring tool, 

15 or an editor. The program that enables a computer to display hypervideo is 
known as the run-time module. 

The authoring tool may be used to define a hotspot, or an object, in a 
region of interest in one or more frames of a video, and then track the hotspot in 
later frames, for example. The hotspot may also be defined in a picture, including 

20 a bitmap. Any subsequent discussion of bitmaps may also be applicable to other 
picture formats, which are subsequently described. Hotspots can be alternatively 
tracked manually and automatically. With the authoring tool, the hotspot may be 
linked to a target, for example, but not limited to, text, audio, or a second video. 
Targets are further described and exemplified below. Thus, for example, when 

25 playing the hypervideo with the run-time module, a user can place a pointer over 
the hotspot in the video with a mouse. When the pointer is placed over the 
hotspot, the cursor may change. Then, by actuating a mouse button, the user 
may launch, or execute, the target. When the mouse button is actuated when the 
pointer is over the hotspot, the cursbr may change again. Multiple targets can be 

30 linked to a hotspot. Because of its ease of use, hypervideo can be used for a 
wide variety of applications, including, but not limited to, interactive television, 
games, tourism and home shopping. 
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2.0 Software Implementation 

Exemplary structures of computer programs for authoring and playing 
hypervideo are respectively shown in Figures 1 A and IB. The computer 
programs comprise a hypervideo environment 1000 including an authoring tool 
5 1001 and a run-time module 1101. The computer programs may be implemented 
with object-oriented software, as exemplified below. The computer programs 
can be executed on a computer, for example, using the Windows 95 operating 
system by Microsoft Corporation (Redmond, Washington). Thus, the programs, 
specifically the authoring tool 1001, may have the look and feel of Windows 95. 

1 0 However, the present invention may be implemented, for example, in other 
systems, such as, but not limited to, televisions, described below. The present 
invention may also be implemented with other operating systems. 

Figure IC illustrates an exemplary computer system 1605 in which one or 
more programs or sub-programs of the hypervideo environment 1000 may reside 

1 5 and be executed. The computer system 1 605 may be any processing system for 
executing one or more programs or sub-programs of the hypervideo environment 
1000, including, but not limited to, personal computers and interactive 
televisions. 

The computer 1605 may include a central processing unit 1610 and 
20 memory 1 620. The processing unit 1610 may be, but is not limited to, a Pentium 
microprocessor by Intel Corporation (Santa Clara, California). The memory 
1620 can be random access memory, disk storage, CD-ROM storage, digital 
video, or versatile, disk (DVD) 1 159, another type of memory or combinations 
thereof The memory may be a video source. Within the memory 1620, the 
25 computer system 1 605 has access to its operating system 1 630 and user software 
1 640. The user software 1 640 can include the authoring tool 1 001 , the run-time 
module 1101, stand-alone modules 1 107, multimedia control interface (MCI) 
driver 1 109, software development kit (SDK) 1111, plug-ins 1 103, and the 
hypervideo project file 1670. 
30 In one embodiment, the computer 1605, or portions thereof, may be 

implemented in an integrated circuit. For example, as illustrated in Figure ID, the 
integrated circuit 1695 can be implemented in a DVD apparatus 1625 including a 
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reader 1627 for reading information from a DVD, operatively coupled to the 
integrated circuit 1695. In another embodiment, the computer 1605, or portions 
thereof, may be implemented in a television 1685, as illustrated in Figure IE, for 
receiving hypervideo television signals. The television 1685 also includes a 
5 receiver 1 687 which may be coupled to the computer 1 605 or portions thereof. 
The computer may permit displaying a hypervideo, for example, on the television. 

The hypervideo project file 1670 is a database used by both the authoring 
tool 1001 and the run-time module 1101. The extension of hypervideo database 
files may be *.OBV. The project file 1670 may be encrypted creating encryption 

10 keys using functions, such as srand and rand. The authoring tool 1001 and run- 
time module 1101 may include components that detect the authoring tool 1001 
version used to create a project file 1670 and thus can read the project file 1670. 
Thus, the authoring tool 1001 and run-time module 1101 may be upgraded 
without project files created by older versions of the authoring tool becoming 

1 5 obsolete or unusable. The authoring tool 1 001 and the run-time module 1101 
will now be successively described. 

?4 ApthorPWgTftol 

The authoring tool 1001 may be used to define hotspots in media, such as 

20 video, and link 1010 the hotspots to targets. Hotspot defmition, for example, 
may be performed substantially in real time. The authoring tool 1001 comprises 
three interconnected sub-programs, the media warehouse 1003, the workshop 
1005, and the project view 1007. Each sub-program may be represented by a 
window on the display of a computer. Furthermore, apparatuses and methods 

25 described below for the authoring tool 1 001 may also be used in the run-time 
module 1 101, or vice versa. The authoring tool 1001 may plug into other editing 
tools, such as Adobe Premier by Adobe Systems Incorporated (San Jose, 
California). 

The authoring tool 1001 may include a registry 1006. The registry 1006 
30 permits each user to save their own authoring tool settings, including the 

placement of windows. The registry 1006, for example, may be implemented 
with the registry system of Windows 95 or Windows NT by Microsoft. 
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A xiser may utilize the authoring tool 1001 to perform the following tasks. 
The user may import media files 1680 into the media warehouse 1003. Each 
imported media file 1680 may result in a new media element 1690 being created 
in the media warehouse 1003. Each media element 1690 may include a reference, 
5 which may be a pointer, to media files 1 680 which could be video, bitmap, sound, 
or text. The pointer may also be directed to non-media, such as a uniform 
resource locator (URL) or an executable file, which are considered to be media 
with respect to media files 1680 and media elements 1690 in this document. One 
or more media files 1680, generally video and bitmap files, may be selected to 

1 0 form a base target. Then, these video and bitmap files may be placed in the 
workshop 1005 so that hotspots may be defined within them. After the hotspots 
are defined, each hotspot may be linked 1010 to one or more other targets, which 
are media elements 1 690. The base target and other targets form a hypervideo 
story board 1116. The aforementioned techniques may be described as nonlinear 

15 authoring 1009. 

The hypervideo data created by manipulating the different sub-programs 
of the authoring tool is stored in a project file 1670. The hypervideo data may 
include, for example, but is not limited to, data pertaining to media elements 
1690, hotspots, targets, and cursors. The project file 1670 can be exported to 

20 video servers, for example, that provide video on a network 1650, such as the 
Internet or an Intranet. Alternatively, it is envisioned that hypervideo data may 
be exported by the authoring tool 1001, for example, into a media file 1680 or 
into a data stream 1660 that is parallel with streams of video and audio 
infomiation. For example, the data may be stored in enhanced video files. 

25 Enhanced video files include hypervideo information in the video file. One form 
of an enhanced video file, such as an AVI file, may only include hotspot 
information. Enhanced video files may also be recognized in the hypervideo 
environment 1000. 

The authoring tool 1005 includes a graphical user interface (GUI) to 

30 facilitate authoring hypervideos. However, a scripting language 1004 can also be 
used to author hypervideos. The GUI includes the following components: 
A Menu bar; 
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A Media Warehouse window - which contains the media elements 1690 
including the references to the media files 1680 in a hypervideo project; 

A Workshop window - in which the user defines the hotspots, and links 
the hotspots to targets. From the Workshop window the user can also open the 
5 Preview, or Cinema, window to review the hypervideo project at any time. 

A Tools window - which holds tools with which the user may define the 
hotspots and link 1 0 1 0 the hotspots to targets; 

A Preview window - with which the user may preview the hypervideo 
project; and 

10 A Project View wmdow - in which the user can view and may also edit 

the logical flow of a hypervideo project. 

The GUI will be subsequently described in further detail. 

2,11 The Menu Bar 

15 The authoring tool may include a menu bar to manipulate the Media 

Warehouse, Workshop and Project View windows. The menu bar may include 
the following commands: 



Command 

File 

New 
Open,.. 

Save 
Save As. 
Revert 



Project Settings 
Base Target 
Default Cursors 
Coordinate Resolution 



Description 

Handles file commands. The file entity is either a 
hypervideo project file 1670 or a media file 1680. 

Creates a new project file 1 670. 
Opens an existing project file 1670. May also preview 
the corresponding hypervideo in a project view window. 
Saves the project file 1670 with its current name. 
Saves the project file 1 670 with a new name. 
Retrieves the last saved version of the project file 1 670. 
This operation closes the current file and opens its 
previously saved version. 

Subsequently described. 
Subsequently described. 
Subsequently described. 
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Command 

Import Media File. . . 

Import From Project. . . 



Import Cursor 

Preview 
Print 

Print Setup... 
Exit 
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Description 

Imports a media file 1680 or folder. This option lets the 
user select a directory and display all media files 1680 in 
that directory. The user can select one, several or all 
media files 1680 in the specified directory. See Figure 2. 
The user may drag and drop files or folders fi-om an open 
import media file dialog into the Media Warehouse 
window. 

Imports one or several media elements 1690 fi'om a 
selected, existing project file 1670. This option lets the 
user utilize previously defined hotspots in the media 
elements 1690. The media elements 1690 are associated 
with the selected project. An imported media element 
1 690 includes all defined hotspots within the media 
element 1690. This command saves the user time when 
defining hotspots. See Figure 2. 
Imports cursors to a hypervideo project. May be 
alternatively accessed by the browse command, described 
later. All cursors that are imported to the hypervideo 
project can be displayed in a cursor window. The ciu-sor 
window can be a simple list view which enables the user 
to import new cursors into the project. When selecting a 
cursor for a hotspot, the user can drag a cursor fi'om the 
cursor window to the Hotspot Properties Sheet. 
Animated cursors may be shown in their animated form in 
the cursor window. The cursor data need not be stored 
in the project file 1670. 

Runs the hypervideo project from its beginning. Also 
may be initiated by the preview button in the Workshop 
window. 

Prints reports of the hypervideo project. The following 
reports can be printed: 

Project Scheme - Illustrates the story board 1116 
created in the authoring tool 1001, 

Project Probabilities 1096 - Calculates and prints a 
report of the probability of reaching each branch in the 
hypervideo. 

Project Listing - Prints a list of all hotspots in the form 
of an identifier (ID) (i.e. text) 

Sets up a printer. Standard dialog for setting up the print 

environment. 

Exits the program. 



Edit 
Undo 



Undoes the last action. 
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Copy 
Paste 

Delete Del 
Select All 
Validate 

Command 

Media 

Partial View 
Detailed View 

Sort by 
View only 

Workshop 
Preview Mode 
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Cuts the selected item and places it in the clipboard. This 
action may be context related. Thus, when the user 
selects this command, the action performed by the 
authoring tool 1001 is related to the item that is selected 
(i.e., hotspot or media element 1690). The Cut command 
may be a multiple selection command, as described 
below. 

Copies the selected item in the clipboard. This action is 
context related. This means that when this command is 
used, the action performed by the authoring tool 1001 is 
related to the item that is selected (i.e., hotspot or media 
element 1690). The Copy command is a multiple 
selection command. 

Pastes the clipboard contents into the specified location. 

See the Cut command about context relation. 

Deletes the selected item whether from the Media 

Warehouse window or the Workshop window. 

Select all items. Applicable both to the Workshop and the 

Media Warehouse windows. 

Validates a hypervideo project by searching for dead 
locks, errors, and minor authoring mistakes. This 
operation may generate a report that can be printed and 
reviewed by the user. Different levels of warnings are 
permitted. 
Description 

Handles commands for manipulating the Media 
Warehouse window. 

View of the media elements 1690 without their details. 
Just the name of the file and a thumbnail are displayed. 
View of the media elements 1690 with all details and a 
header for each of the categories. The user can actuate a 
header to sort the media elements 1690 according to the 
selected category. 

Sort the media elements 1690 in the Media Warehouse 
window by a specified category. 
View only a certain type of media file 1680 (e.g., video, 
sound, bitmap, etc.). 

Manipulates the Workshop window. 
Enters preview mode in which the hypervideo project 
plays and permits navigation through the hypervideo 
with hotspots. 
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Play / Pause 



In 

Out 

Home 

Define New Hotspot 

Set Target 
Options 
Command 

Settings 



Plays and Pauses the video displayed in the Workshop 
window. These commands appear depending on the 
state of the Workshop window. If Workshop window 
is playing a video, the command shows a Pause button. 
However, if the video is paused, the Workshop 
window shows a Play button. 

Marks a currently displayed frame as default IN for the 
media, such as video. 

Marks the currently displayed frame as default OUT for 
the media, such as video. 

Goes to the IN frame of the video in the Workshop 
window. 

Defines a hotspot of the selected shape (e.g.. Elliptic, 

Rectangular, Triangular, or Polygon). 

Enters the Set Target mode, as described below. 

Authoring Tool options. 

Description 

May pop up a properties sheet with settings for all 
windows and the hypervideo project. The properties 
sheet may be used to define run-time details, including 
the size and position of the Preview window, as well as 
assign the base targets of the project. The base targets 
may play when the project is started. To define base 
targets, the user may drag video media elements, for 
example, from the Media Warehouse 1003 to the 
Targets, or Links, page of the properties sheet. Also, 
the properties sheet may permit controlling automatic 
tracking, hotspot colors and patterns (typically 
crosshatched), maximum number of polygon vertices 
and other workshop parameters. A hotspot may have 
three states: Selected, Deselected and Interpolated. 
The user can choose colors and patterns for each state. 
The maximum number of polygon vertices limits the 
number of vertices that may be used to define a hotspot 
in a freehand mode. 



Window 
Media 
Workshop 
Tools 
Close 
Close All 



Controls the windows of the authoring tool 1001. 
Makes the Media Warehouse window appear. 
Makes the Workshop window appear. 
Makes the Tools window appear. 
Closes the selected window. 
Closes all windows. 
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Selection of specific menu bar commands may launch dialogs for 
importing media files. Specifically, dialogs are launched when placing media 
elements 1 690 into the media warehouse 1003 with the import media file and 
5 import firom project commands. These dialogs are described below. 

Itaiport Media Fik 

In order to start building a hypervideo project, the user may import media 
files 1680 into the media warehouse 1003. Figure 2 is an illustration of an 
1 0 exemplaiy Import Media File dialog 200 1 . The Import Media File dialog 200 1 is 
launched upon the selection of the import media file conmiand. The Import 
Media File dialog 2001 permits the user to view media files 1680 and import 
corresponding media elements 1690 into the media warehouse 1003. The user 
can import media elements 1690 in the following ways: 
15 1 . Import a media element 1 690 corresponding to a media file into the media 
warehouse 1003. 

2. Import media elements 1690 corresponding to media files 1680 in a 

directory or file folder into the media warehouse 1 003 . 
This dialog 2001 may use a multiple selection list. The Import Media File dialog 
20 2001 may permit several files or an entire directory to be selected. 

When actuating a Preview button 2003, the dialog may show a preview of 
the selected file, with the ability to execute, for example, a video, audio or bitmap 
media. Media files 1680 and folders may also be dragged and dropped, for 
example, firom Explorer by Microsoft and other file managers. 

25 

Impon from Prpjgct 

Figure 3 is an illustration of an exemplary Import From Project dialog 
3001 . The Import From Project dialog 3001 permits the user to view and import 
media elements 1690 from a hypervideo project file 1670 into the media 
30 warehouse 1003. A preview window may be opened to display both the 

hypervideo project and corresponding media files 1680 by actuating the Preview 
button. In the preview window, hotspots may be shown. The Import From 
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Project dialog 3001 includes features of the Import Media File dialog 2001 
described above. 

2.n Maw TqqI Bar 
5 The hypervideo authoring tool 1001 may have a main tool bar. The main 

tool bar may float and can be closed. The main tool bar functions may include: 
New, Open, Save, Import Media File, Print, Preview, Copy, Cut, Paste, Media 
view formats. Options | Settings, Window Tiles, and Help. 

10 2,13 Media Warehouse 

The Media Warehouse 1003 may contain media elements 1690 that are 
used to form a hypervideo project. Each media element 1 690 may include a 
reference to a media file 1680 which could be video, bitmap, sound, or text. The 
pointer may also be directed to non-media, such as a imiform resource locator 

1 5 (URL), a hypertext markup language (HTML) file, or an executable file, as 

described above. For example, the URL may point to HTML file. The video and 
bitmap media elements may include hotspots that are linked 1010 to targets, such 
as other media files 1680. 

20 The Media W arehouse Window 

The Media Warehouse 1003 may be represented by a window on a 
display on a computer or television, for example. The Media Warehouse window 
displays a thumbnail of the contents of a media file 1680 pointed to by a media 
element 1690. The Media Warehouse window may permit the user to view the 

25 media elements 1690 in one of several views, including: 

1 . Symbolic view - in which, for example, only thumbnail illustrations of the 
contents of media files 1680, referenced by corresponding media elements 1690, 
are displayed; and 

2. Detailed view - in which detailed media element 1690 information is 
30 displayed. 
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Each media element 1690 has its own identification (ID) section in the 
Media Warehouse window. As shown m Figure 4, the media element's ID 
section, or icon, 4001, may consist of: 

A thumbnail illustration 4007 of the contents of a media file 1 680 pointed 
5 to by a media element 1690. The thumbnail illustration 4007 is a picture 

in the case of bitmap or video media files, and a waveform in the case of 

sound media files. For video media, the picture may show one frame, 

presumably the first fi^e, of the video media file. 

An icon specifying the data type of the media file 4005 for any view that 
10 does not require a specification of the content of the media file 1680, but 

rather the data type and name of the media element 1690. 

Textual data that describes the media file 1680. 

In the Media Warehouse window, if a media element 1690 includes at 
least one target, the right side of the media element icon 4001 may be decorated 
15 by a token 4003, or link button. This token 4003 indicates that the media 
element 1690 is linked 1010 to a target, such as a media file 1680. The token 
4003 may also be a button. Upon actuating the mouse button when a pointer is 
over the token 4003, the user may be presented with a pop-up window that 
shows all targets that are linked 1010 to the selected media element 1690. The 
20 user may drag the pointer over a target and actuate the mouse button. Then, the 
selected target may scroll up or down to the top left position in the Media 
Warehouse window. The pop-up window may then show any targets linked 1010 
to the selected target. This method provides an easy and intuitive method to 
navigate through the files in the Media Warehouse window. The linked target 
25 pop-up window may be closed by actuating the mouse button when the pointer is 
over the token 4003 of the selected media element. Figure 4 shows an exemplary 
media element icon 4001 for a video in the Media Warehouse window. 

Alternatively, the token 4003 may be implemented as an icon, and not a 
button. As a resuh, the Media Warehouse window can be implemented as a 
30 common list view control. 

The authoring tool 1001 can show targets linked 1010 to hotspots in a 
manner that is more easily understood by a user. Hotspots, rather than links 
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1010, can be viewed by manipulating the media element icon 4001 . For example, 
when the pointer is positioned over a media element icon 4001 and the right 
mouse button is actuated, a menu of hotspots, rather than linked targets in the 
selected media element, is displayed. Then, the linked targets of each hotspot 
5 may be displayed. These techniques will subsequently be described. 

The authoring tool 1001 supports the 8, 16, 24, and 32 bit color modes. 
Thus, thumbnails and previews may be displayed in the color mode of the 
corresponding media file 1680, or altematively, in the 24 bit color mode. 

10 Relative Path 

The authoring tool 1001 may create a project file 1670 that defines the 
paths of media files 1680, corresponding to media elements 1690 in a hypervideo, 
relative to the path of the project file 1 670. The common portion of the paths of 
the project file 1670 and media file 1680 is referred to as the static path 1 163. 

1 5 The unique portion of the media file 1 680 path is referred to as the relative path 
1161. The project file 1670 and media files 1680 may be stored locally or 
remotely 1 139. For example, the project file 1670 and media files 1680 may be 
moved to a different memory unit, such as disk drive, computer, or computer 
network, while keeping the relative path 1 163 the same, then the project file 1670 

20 does not require editing for the hypervideo to be performed. This will enable 
altering the drive letter or even the base path from which the hypervideo project 
is executed. 

Tool Tips 

25 A tool tip can be activated and display information about a media element 

1 690 when a pointer is positioned over a media element icon 4001 in the Media 
Warehouse window. The tool tip, for example, may show the full or relative path 
1 161 of the corresponding media file 1680. The tool tip may include other 
information, such as the type, and logical and physical size of the corresponding 

30 media file 1680. 
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Media Warehouse W indow Controls 

The Media Warehouse window may be implemented using a custom list 
view control from the Microsoft Windows 95 common controls. The list view 
may show the media elements 1690 in different views (i.e., with details or icons), 
5 and permits user manipulations of the lists, such as sorting and reordering. Each 
media element 1690 of the Media Warehouse window list corresponds to a media 
file 1680, such as a bitmap. The Media Warehouse wmdow list is associated with 
an image list control that includes a thumbnail illustration 4007 and a description 
of the type of the media file 1680. The text of the Media Warehouse window list 
1 0 describes file parameters and are displayed to the user in a detailed view. 

The Media Warehouse window can be manipulated in the following ways. 
Using the list view, the user can, for example: 

• Resize the list to any size, and create a column like list or a scattered list. 
Change the order of the media elements 1 690 by dragging and dropping 

1 5 media element icons 4001 corresponding to the media elements 1 690, 

inside the list. 

In the detailed view, sort media elements 1 690 by category, including 
media file 1680 type and size. Sorting may be perfomied by actuating the 
mouse button when the pointer is over the desired list view header. 
20 • Actuate the alternate mouse button to obtain a list of all targets linked 
1010 to, or all hotspots in, a specific media file 1680. By selecting a 
specific target, for example, the user can access the corresponding media 
element 1 690 in the list. 

Double click the mouse button when the pointer is over a media element 
25 1 690 to preview the corresponding media file 1 680. This option may also 

be activated with the alternate mouse button. 

• Drag and drop media elements 1 690 into other windows of the authoring 
tool 1001, such as the Workshop window. For example, when a media 
element icon 4001 is dropped into the Workshop window, the respective 

30 media file 1680 is opened and, for example for a video, its first frame is 

displayed. 
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The lists of the Media Warehouse window may support full multiselect 
operation for the delete and move functions. 



10 



15 



20 



25 



30 



Altgffi&te Mouse Puttop 

The alternate mouse button can be actuated to perform the following 
functions, depending on the media file 1680 type of the media element 1690 that 
is selected. 

Preview - open a viewer for that media file 1 680 type. Videos that are 
linked 1010 may not display any hypervideo capabilities in the 
viewer. 

Send to Workshop - The project file 1670 will be opened in the Workshop 
window (see subsequent description). 

Links ► - May open a list of targets (e.g., media elements 1 690) that are 

linked 1010 to the selected media element 1690. Choosing one of 
the linked targets will move the list to the selected target. 
Alternatively, a list of hotspots, not necessarily linked 1010 to 
targets, in the selected media element 1690 is displayed. Selecting 
one hotspot will display a list of the targets linked 1010 to the 
hotspot. The latter option enables fast and intuitive navigation 
through component media elements 1690 of a hypervideo project. 

Insert...- Imports a new media file 1680 and creates a corresponding media 
element 1690. The corresponding media element 1690 is placed 
before the selected media element 1690 in the list view window. 
The new media file 1680 can be previewed when performing the 
Insert function. 

Delete - Deletes the selected media element 1690 from the Media 

Warehouse window. 
Properties. • .- Permits viewing and editing of media properties of the 
selected media element 1690. 

Editing Options 
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The media list of the Media Warehouse window may support the 

following editing commands: 

• Undo - Upon deleting or moving a media element 1690 in the list. 

• Delete - Permits multiselect deleting. 
5 • Cut 

Copy 
Paste 

• Select All - Selects all media elements 1690 currently in the list. 

10 Viewing Options 

The user can enable the Media Warehouse window to display only media 
elements 1690 corresponding to certain types of media files 1680, such as only 
video or sound files. Furthermore, the media elements 1690 in the Media 
Warehouse window can be manually arranged by the user. Typically, though, the 

1 5 media elements 1 690 are automatically arranged. 

Media PrQpgrtiigs Sheet 

Properties for each media element 1690 may be shown in a Media 
Properties Sheet. The Media Properties Sheet may be a Win32 properties sheet 

20 with tabs that represent each page in the properties sheet. The Media Properties 
Sheet may include a Details, or General, page. The Details page includes general 
file information relevant to the specified media type. Figure 5 illustrates an 
exemplary Details page 5001 of the Media Properties Sheet 5003. 

The Details page 5001 may include the file name and path of the media 

25 file 1 680. The media file 1 680 name and path may be too long to be viewed on 
the Details page 5001. Therefore, only the media file 1680 name may be 
displayed in the Details page 5001 . The path may be displayed in a tool tip when 
the user moves a pointer over the file name. The Details page 5001 also may 
include default cursors of the media element 1690. 

30 The Details page 5001 may also incorporate general file data such as size 

5005 and location 5007, together with creation and modification dates 5009, 
501 1 . One section of the Details page 5001 may include data pertaining to the 



-WO 99/10822 



PCT/US98/17444 



23 

type 5019 of media file 1680 to which the corresponding media element 1690 and 
Media Properties Sheet 5003 are associated. Relevant data for different media 
file types 5019 is described below: 

• Video - the pertinent data may include the length in seconds 501 3, and the 
5 audio and video formats 501 5, 501 7. Another parameter is the 

manual tracking speed for the video. Each video may have an 
optimal manual tracking speed. The user may control the manual 
tracking speed with a slider. 

• Bitmap - the pertinent data may include the pixel size of the bitmap, number 
10 of colors, and compression type if applicable. 

• Audio - the pertinent data may include the audio format and length in 

seconds. 

The second page of the Media Properties Sheet 5003 may be a Preview 
page. Figure 6 illustrates an exemplary Preview page 6001 for a video. Videos 

15 may, for example, be displayed in either 320 by 240 or 160 by 120 pixel areas. 
Bitmaps may be viewed in a fixed pixel area of 160 by 120. Audio emissions and 
video frame display may be controlled with a slider 6007 to manipulate the audio 
and video file advances, and also possibly to display a corresponding waveform of 
the audio file. Text may be shown in a scrolled window. 

20 The third page of the Media Properties Sheet 5003 may be the Targets, or 

Links, page. Figure 7 A illustrates an exemplary Targets, or Links, page 7001. In 
the Targets page 7001 , the user can view a list of targets 7003 that are linked 
1 010 to hotspots in the selected media element 1690. The Targets page 7001 
also includes an Add Target button 7005. The user actuates the Add Target 

25 button 7005 and holds the mouse button down just as if the user were clicking the 
video hotspot in the Set Target mode. The cursor changes to the Set Target 
cursor, and the process of setting a target begins as v/ill be subsequently 
described. Other methods for linking 1010 targets to hotspots may be used. 
The Media Properties Sheet 5003 may also have a Hotspot page 701 1 , 

30 exemplified in Figure 7B, that illustrates the hotspots in a selected media element 
1 690. The Hotspot page 701 1 may display a list of hotspot locations in 
thumbnail illustrations 4007. For a video, the thumbnail illustration 4007 may be 
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the first frame in which the hotspot appears. Placing the pointer on a thumbnail 
illustration 4007 of a selected hotspot in the list and actuating, for example, by 
double clicking, the mouse button may cause a corresponding Hotspot Properties 
Sheet to be opened. The Hotspot Properties Sheet 7101 may include the Targets 
5 page 700 1 , which, in this case, may only display a list of targets 7003 linked 1010 
to the selected hotspot. 

The Hotspot Properties Sheet 7101 may also include a General Properties 
page 7103 exemplified in Figure 7C. The General Properties page may include 
the hotspot name 7103, type of shape 7105 defining the hotspot, media file name 
10 in which the hotspot is located 7107, the range of frames in a video media file in 
which the hotspot 8003 is located 7109, and cursor types. The cursor types 
include the cursor displayed when the pointer is over the hotspot 7111, and the 
cursor displayed when the mouse button is actuated when the pointer is over the 
hotspot 71 13. 

15 

2,14 The Workshop 

The Workshop 1005 may be used to define hotspots 1023 and create links 
1010 between hotspots and targets. Hotspots, for example, may be created in 
videos and bitmaps. The project may be viewed in the Preview, or Cinema, 

20 window. The Workshop 1005 is displayed in a Workshop window that may have 
a toolbar window attached to it. The Workshop window changes in accordance 
with the tool selected from the Tools window. The Workshop window has an 
interface that permits selecting a specific fi-ame in a video, and forwarding or 
reversing the displayed video fi*ame. When displaying bitmaps, video fi-ame 

25 position controls are disabled. Figure 8A illustrates an exemplary Workshop 
window 8001 displaying a fi-ame 8013 of a hypervideo. 

In the Workshop window 8001, the user may define and edit hotspots 
8003 in video, for example, and link 1010 the hotspots 8003 to targets 7003. 
The user can preview the hypervideo project by toggling the Preview, or Cinema, 

30 switch 8005 to the ON position to pop up the Preview window. The Preview 
switch 8005 may, however, be a button. 

The Workshop window 8001 may include the following features items: 
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Play button 8007 - plays and pauses a video. The play button, as for many 

player interfaces, turns into a Pause button while the video 
plays. 

Home (or In) button - displays the IN frame (default is the first video frame). 
5 End (or Out) button - displays the OUT frame (default is the last video frame). 
Slider 6007 - permits the user to select a desired frame of a video. The frames 

of the video are displayed as the slider is moved. 
Frame 801 1 +/- buttons - respectively moves one frame 801 3 forward (+) and 

backward (-), and is useful for pinpointing a 
10 particular frame. 

Scale Information - gives the user frames/time information. 
Preview switch 8005 - opens the Preview window. 

The Workshop window 8001 may serve as a video viewer. With the 
Tools window, a user can also use the Workshop window 8001 to define 
15 hotspots 8003 and links 1010 to targets 7003. The Preview and Tools windows 
will now be described. 



The Preview Window 

The Preview window is a hypervideo player in the authoring tool 1001 . 
20 In the Preview window, the user can execute targets by activating, or actuating, 
hotspots in the hypervideo, for example. The Preview window 8101, illustrated 
in Figure 8B, may be similar to the Workshop window 8001 except for the fact 
that it is a video viewer that may not permit editing. In this case, the IN/OUT 
and Preview 8005 switches are absent. 

25 

2,141 Hotspot Definition 

Hotspots 8003 are defined in media, such as video or bitmaps, in the 
Workshop window 8001 with the methods described below. 
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The Tools Window 

The Tools window 9001 includes one or more dockable toolbars that are 
used to define hotspots 8003 and link 1010 hotspots 8003 to targets 7003. 
Figure 9A illustrates an exemplary Tools window 9001 . Typically, the Tools 
5 window 9001 may be attached to the Workshop window 8001 . 

The Tools window 9001 can dock on any side of the Woricshop window 
8001 . The Tools window 9001 is a toggle toolbar. Thus, only one button (i.e., 
mode) can be used at any instance of time. The Pick Hotspot button 9003 may 
be the default selection mode of the Tools window 9001 . Each selected mode 
1 0 changes the behavior of the Workshop window 8001 . The Tools window 9001 
modes will now be described: 

Pipk Hotspot 

The Pick Hotspot mode 9003 permits the user to select and manipulate a 
1 5 defined hotspot 8003 . In this mode, the user can: 

Select a hotspot 8003 or a group of hotspots. 

• Reposition a hotspot 8003 in the current frame. 
Resize a hotspot 8003 in the current frame. 

• Double click on the hotspot 8003 and view its properties. 
20 • Press Del (Delete) and delete the hotspot 8003. 

Holding down the SHIFT key while moving the Workshop window slider 
6007 marks a selection of frames. Any action that is performed in the Pick 
Hotspot mode 9003 will be executed throughout the range of selected frames. 

25 Ppfme Hotspot 

The Define Hotspot mode 1025 permits definition, or segmentation, of 
new hotspots 8003. When selecting the Define Hotspot mode 1025, all 
previously selected hotspots 8003 are deselected. The hotspot is then defined by 
drawing the desired shape 91037. The user may select a primitive shape 1039 

30 from the Tools window 9001 shown in Figure 9A. The primitive shape 1039 may 
be, but is not limited to, an ellipse 9009, a triangle 9007, or a rectangle 9005. 
The primitive shape 1039 may be default-sized. However, a polygon 1041 may 
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also be selected. The user may then place the selected shape in the media in the 
Workshop window 8001 . After the shape has been placed, the Tools window 
9001 may automatically switch to the Pick Hotspot mode 9003. Then, the user 
can manipulate the size and position of the selected shape to define the hotspot 
5 8003. Hence, the hotspot's geometric form may be defined. 

Hotspots 8003 having the shape of a polygon 1041 can be defined, for 
example, in one of three ways: 

(1) With a magic wand 9011 , described below; 

(2) With a polygon drawing tool with which the user draws lines between 
1 0 polygon vertices; and 

(3) With freehand drawing, the user draws the hotspot 8003 in freehand fomi 
and the Workshop 1005 calculates the optimal number of vertices limited 
by the maximum number of vertices parameter. 

Hotspots 8003 having the shape of a polygon can be re-sized using a 
1 5 bounding rectangle or by moving each vertex of the polygon. The user can select 
either mode in the Workshop window 8001 . 

Upon its creation, a new hotspot 8003 may be given a default name, 
which may be changed later. The hotspot name may be added to a list of defined 
hotspots 8003 in the Workshop's Tools window 9001, which may also include a 
20 Hotspot toolbar. If the user double-clicks the mouse button when the pointer is 
over the hotspot 8003, the Hotspot Property Sheet 7101 may be displayed. In 
the Hotspot Property Page, the user can assign a different name to the hotspot 
8003, and the cursors that are to be used when the pointer is positioned over the 
hotspot 8003 or when the hotspot 8003 is activated. 
25 Furthermore, the Hotspot toolbar allows the user to select a hotspot 8003 

and display the first frame in which that hotspot 8003 appears in a video. The 
user can also open the Hotspot Property Sheet 7101 from the Hotspot toolbar as 
well as delete the currently selected hotspot 8003 and all of its occurrences. 
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Magic Wand 

The Magic Wand 901 1 is used to define a hotspot 8003 automatically 
1033, rather than with shape drawing 1037. When the user points the Magic 
Wand 901 1 at a pixel in a region of interest, which may be defmed as a hotspot 
5 8003, and actuates a mouse button, the Magic Wand 901 1 searches for edges 
around the pixel 1049 that define a new hotspot 8003 or modify a pre-existing 
hotspot 8003. The Magic Wand 901 1 may use either the rays or flood-fill 
algorithms 1051, 1053. With the flood-fill algorithm 1053, flooding can be 
performed based upon either the Red-Green-Blue (RGB) or luminance values of 

1 0 pixels. The flooding may be performed within a tolerance range of RGB color or 
luminance values around respectively an initial RGB color or luminance value of 
the pixel. As a result, all pixels surrounding the initial pixel with a RGB or 
luminance value within the tolerance range will be flooded. The tolerance range 
can be modified by the user. Flooding may require a line-table hotspot type. 

15 Hotspots 8003 may be added or subtracted from one another to create 

hotspots 8003 of complex shapes. New hotspots may be created by uniting, or 
adding, multiple hotspots with one another, subtracting one hotspot 8003 from 
another, or by intersecting multiple hotspots. The flood-fill algorithm 1053 may 
also permit incremental hotspot definition by adding newly flooded areas to areas 

20 that are already flooded. The flood-fill algorithm 1053 may also permit flooding 
out areas that are already flooded. The user can interact with the flood-fill 
algorithm 1053 with a floating menu or the alternate mouse button. 

With the rays algorithm 1 05 1 , imaginary rays in different directions are 
launched either outwards fi-om an initial point, or from the hotspot 8003 borders 

25 inwards towards the initial point. The rays algorithm 1051 searches along the 
rays for intersections with the region of interest borders, or edges, that will define 
the hotspot 8003. The edge points along the rays will be scored using different 
edge scoring algorithms. Upon completing score, a point on the ray may be 
selected that either has the best score or is the first point to have score that 

30 exceeds a threshold value. When intersection point is found, the intersection 
point is selected as a new polygon vertex. The number of rays used to define the 
hotspot 8003 can be modified by the user. The Magic Wand 901 1 can also use 
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edge-enhancing filters 1043 using threshold 1045 and spatial derivative 1047 
techniques, prior to using the edge detection algorithms. 

Zoom Optipns 

5 The Workshop window 800 1 may permit zooming within a ftame to 

define hotspots 8003 more accurately. This feature may be particularly useful for 
creating a hotspot 8003 in the shape of a polygon 1041 with, for example, the 
magic wand 901 1 or freehand drawing. 

10 MCTging Hotspots 

The user can merge multiple hotspots 8003 that may have different shapes 
into a single hotspot 8003. Hotspots 8003 can be merged over a selected range 
of one or more frames in a video. Additionally, the user can remove a hotspot 
8003 from a group of merged hotspots. The user interface for merging hotspots 

1 5 8003 can be incorporated into either the Workshop window 8001 or the menu 
bar. A dialog may be used to enter data pertaining to merged hotspots. 

Iffterpol^tioiff 

Interpolation 1027 permits a user to quickly create hotspots 8003, for 
20 example, in a range of video frames. One embodiment of creating a hotspot 8003 
with interpolation 1027 will now be described. First, the user defines the hotspot 
8003 in a selected initial, or first, video frame, and presses the Interpolate From 
button 9002. As a result, the hotspot's color changes. Next, the user defines the 
hotspot 8003 in a selected last, or second, video frame, after the initial video 
25 frame, of the desired range, and presses the Interpolate End button 9004. As a 
result, the hotspot 8003 is defined by interpolation of size and position in the 
desired range of video frames. In one embodiment, interpolation 1027 is 
performed with linear interpolation. The Interpolate From and End button 9002, 
9004 may, for example, be part of the Tools window 9001 . 
30 The user can also define the hotspot 8003 in one or more selected 

intermediate frames between the initial and last video frames. As a result, the 
hotspot is defined by interpolation, such as linear interpolation, between each 
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successive selected video frame. Also, a hotspot 8003 may be defined by 
interpolating between two different shapes, for example, in two different frames 
of a video. 

S 2,142 Hotspot Tracking 

Upon defining a hotspot in an initial frame of video, the hotspot can be 
tracked 1029 through successive frames. Tracking may be performed manually 
1035 or automatically 1061 . Hotspots can be tracked while the video is playing 
in reverse. 

10 

Manual Trapkiipg 

Manual tracking 1035 permits the user to control the placement and size 
of the hotspot 8003 in each frame of the video. An exemplary method for manual 
tracking 1035 will now be described. First, the user selects a hotspot 8003 by 

15 depressing a (e.g., left) mouse button. Then, while depressing the (e.g., left) 
mouse button, the user actuates, such as by clicking, the (e.g., right) altemate 
mouse button. The video will then start playing frame by frame at the specified 
tracking speed set in the video's Media Properties Sheet. Manual tracking 1035 
is performed as long as the user keeps the mouse button depressed. The user 

20 toggles 1057 between executing and pausing manual tracking 1035 by 

respectively depressing and releasing the altemate mouse button. The user can 
manually move, or slide, 1055 the hotspot 8003 position 1059 about the 
Workshop window 8001 by moving a mouse, for example, while the video is 
playing. The user can resize 1059 the hotspot 8003 with the arrow keys while 

25 manual tracking 1 035 is being performed. 

Upon releasing the mouse button, the hotspot 8003 will remain selected. 
However, the Workshop window mode changes automatically to Pick Hotspot 
mode 9003. Inthismode, the user can resize and move the hotspot 8003. Also, 
the user can then continue to track the hotspot 8003 either manually 1035 

30 (described above) or automatically 1061 . 
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The user can freely switch between manual and automatic tracking. If the 
user wants to redefine an existing hotspot, the user can select the Pick Hotspot 
mode 9003 and again track, manually and/or automatically, the selected hotspot. 



5 Ampgiatic Tm}^m 

Automatic tracking 1061 is activated from the Tools window 9001. A 
hotspot 8003 in an initial frame must be selected to be tracked. Upon automatic 
tracking 1061 activation, a new view window is opened and the tracking results 
are displayed there. Utilizing Win32 multithreading, for example, multiple 

1 0 hotspots 8003 can be simultaneously tracked in separate tracking windows. The 
hotspot 8003 may then be defined in succeeding fi-ames until the automatic 
tracking 1061 is halted. When the automatic tracking 1061 is halted, the view 
window may close and the Workshop window 8001 is updated with the frame at 
which the automatic tracking 1061 halted. 

15 Automatic tracking 1061 of a hotspot 8003 is accomplished with image- 

processing algorithms. Both the location, or position, and size of a hotspot 8003 
may be automatically tracked 1061 over a range of frames in a video. Location 
tracking 1073 identifies and follows any changes in the location of the tracked 
hotspot. Size tracking 1069 identifies and follows any change in the size of the 

20 region of interest corresponding to the hotspot 8003 . Size tracking 1 069 may 
include a resizing algorithm 1071 to efficiently create resized copies of the frame 
or the region of interest, which may be defined by a bounding rectangle. Size 
changes are usually due to a change in perspective like zooming. Size tracking 
1069 can be disabled in a Tracking Property Sheet 9101, exemplified in Figure 

25 9B, for example, with a flag 91 03. 

Automatic tracking 1061 is performed with tracking algorithms 1063, 
which may include methods of selective enumeration 1065 and scoring 1077. 
Selective enumeration 1065 may be performed for each frame to determine the 
new position and size of the hotspot 8003, described above. 

30 Selective enimieration 1 065 may be performed for each fi^e with a 

steepest, or gradient, descent algorithm 1067 to postulate hotspot 8003 size and 
position. Selective enumeration 1065 usually avoids enumerating all possible 
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locations and sizes, thus increasing the speed of the automatic tracking 1065 
process. Using the steepest descent algorithm 1067, a path will be found to the 
hotspot position and size having the best score usually without having to 
enumerate all possible hotspot 8003 positions and sizes. This technique is 
5 efficient for determinmg hotspot 8003 size and position that greatly differ from 
the hotspot 8003 size and position in a previous frame. 

Automatic tracking 1061 may be stopped and tracking failure may be 
declared if the selective enumeration 1065 reaches a maximum offset without a 
minima being found by the gradient algorithm 1067, or if the best score found 

10 does not exceed a threshold beyond the average score of all enumerations. To 
detect when a hotspot 8003 exits a frame, the part of the shape still in the frame 
may be enumerated with a boundary compensation algorithm 1075. 

For each enumerated size and location, scoring 1 077 is performed to 
determine hotspot matching 1079 by correlating the postulated hotspot in the 

15 new size and position with the hotspot 8003 in the previous frame, and to 

recognize tracking failure. Hotspot matching 1079 may measure the correlation, 
or similarity, of two hotspots in the following way. Shape iteration 1085 may be 
used to enumerate efficiently the respective pixels in the postulated hotspot and 
the hotspot in the previous frame, and to calculate the sum of the differences 

20 between each set of two corresponding pixels in the postulated hotspot and the 
hotspot 8003 defined in the previous frame. A smaller sum indicates a high 
similarity between the postulated hotspot and the hotspot 8003 in the previous 
frame. The difference can be measured either by calculating the luminance delta 
1083 of the corresponding pixels, or by calculating the RGB maximum delta 

25 1 08 1 , otherwise known as the maximum norm of the RGB delta values, of the 
corresponding pixels. 

To recognize tracking failure, a simple scoring average may be calculated 
and compared against a best score. If the best score does not differ dramatically 
from the average score, the automatic tracking 1061 process will declare a 

30 failure. However, normalized scoring using variance and weighted score 

averaging can also be used. Additionally, the tracking process can be improved 
by filtering each frame. For example, smoothing or four pixel filters can be used. 
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Special filters can be assigned to specific tracking algorithms 1063 and scoring 
1077 methods. 

Scoring efficiency can be enhanced by using the signature 1087 of the 
tracked hotspot. The hotspot signature 1087 may be a characteristic of the 
5 hotspot 8003 m the first firame the hotspot 8003 was defined. For example, the 
signature 1087 can be the offset between the center of mass 1089 and the 
geometric center of the hotspot 8003. The RGB or luminance content of each 
pixel is used to determine the mass of each pixel. To calculate the center of mass 
1089, the hotspot's pixels may be enumerated using shape iteration 1085. The 

10 benefit of using a signature 1087 is that the score may be calculated by comparing 
data fi-om the current fi-ame with the signature of any previous fi*ame. Hence, the 
pixel data of the entire previous fi'ame need not be stored. 

Smart scoring can also be used. With smart scoring, the scoring system 
learns from each tracking step. For example, relying upon temporal coherence 

1 5 1 086 of a moving hotspot, the hotspot motion can be evaluated and new 
suggested positions and sizes for the hotspot 8003 may be estimated in 
accordance with the motion trend of the hotspot trends. 

Polygons can be automatically tracked 1061 using different techniques, 
including those described below. First, the boxmded rectangle around a polygon 

20 can be tracked, and the polygon moved, as is, to the new tracked location. 

Second, the magic wand 901 1 algorithms can be used to adjust the polygon shape 
after tracking, as described above in the first technique. A third alternative is to 
use a shape iterator 1085, discussed above, for a polygon, and an inflating or 
resizing algorithm 1071 to track and directly score the size and position of the 

25 polygon. 

?443 Targets 

The user can create a hypervideo that is nonlinear 1009 by linking 1010a 
hotspot in a source media to one or more targets 7003. A target 7003 may be a 
30 playing instance of a media element 1690 including certain parameters 

exemplified below. Nonlinear authoring 1009 may permit source media and 
targets to be executed and terminated in a variety of ways, as will subsequently be 
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described. The hypervideo includes one or more base targets, such as a video 
target, that are executed when the hypervideo commences. Targets 7003 may, 
for example, be media elements 1690 or static targets 1022. Media element 1690 
targets may include, but are not limited to: 
S 1. Video 1011 - may be used as hotspot containers from which the user can 

hyperlink to media. Video targets 101 1 (i.e., video media 
files) are not restricted to any specific format. For 
example, the hypervideo environment 1000 supports video 
formats including, but not limited to. Video for Windows 
1 0 (AVI), QuickTime (MOV), motion JPEG and MPEG I and 

II (MPG) files. Video targets 1011 may, for example, be 
in any format for which an MCI driver is available. 
2. Sound 1015 - may be used, for example, as a connecting segment 

between two videos. A sound target 1015 can be a vocal 
1 5 announcement when the user actuates a mouse button 

when the pointer is over an image, for example, of a 
person. Supported soimd target 101 1 (i.e., sound media 
file) formats include, but are not limited to, WAV, AIF, 
RMI and MID. 

20 3. Picture 1013 - may be used as hotspot containers from which the user can 

hyperlink to media. Supported picture target 1013 (i.e., 
bitmap media file) fomiats include, but are not limited to, 
DIB, BMP, GIF, PCX, TIFF, JPEG and PICT. 
4. Text 1017 - may be used, for example, as a connecting segment 

25 between two videos. Text targets 1017 may appear in 

boxes, such as message boxes. Text targets 1017 can be 
in, but are not limited to, a TXT or RTF file format, or 
stored directly in the project file 1670. Text targets 1017 
may be displayed with bitmaps also. 

30 5. Executable 1021 - may be used to launch an executable program (e.g., 

EXE or BAT file) as a target. 
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6. HTML File 1021 - may be opened with a browser (e.g., Internet 

browser for the World Wide Web) or an OLE 
control for viewing an HTML file. 

7. URL Addresses 1021 - may be targets created with the authoring tool 1001 
5 and stored as text. Upon activation of this target 

7003, a browser, such as an Internet browser, may 
be laimched to access the URL. 

8. QuickTime VR 1021 

9. Dialogues 1021 

0 

Targets may also include, but are not limited to other applications and processes, 
commimications links to other computers, function menus, high definition 
television signals, and virtual reality environments. 



1 5 A static target 1 022 is not a media element 1 690, but controls a media 

element 1690 in a hypervideo project. Static targets 1022 include: 

1 . Back Target - Goes back to the media element 1 690 that launched the 

target 7003, 

2. Exit Target - Exits the hypervideo, closes the video window, and returns 
20 control to the system or application that started the 

hypervideo player. 

3. Pause Target - Pauses the hypervideo and causes a play button to pop 

up that enables the user to continue playing the 
hypervideo by actuating the button. 
25 Other targets 7003, extemal to the hypervideo authoring software, may 

also be accessed. Extemal targets may be accessed through a DLL file complying 
with API requirements to permit an open architecture for the hypervideo 
environment 1000. 

Targets 7003, such as media, can be stored on servers 1655 on local or 
30 wide area networks 1650, 1115. Specifically, media, such as video, can be stored 
on high capacity storage devices such as, but not limited to, digital video, or 
versatile, disks (DVDs) 1 159. 
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Linking 

A user may link 1010a target 7003 to a hotspot 8003 by the following 
method: 

1 . Actuate the mouse button when the pointer is over the Set Target button 
5 901 5 in order to enter the Set Target mode. The cursor changes to 

indicate entrance to the new mode. 

2. Select a hotspot 8003 to which the target 7003 will be coupled with the 
new cursor while not releasing the mouse button. The Set Target 
movement is a drag-and-drop action. 

10 3 . While continuing to hold down the mouse button, drag the new cursor 

onto the Media Warehouse window and the selected media element 1690 
that is to be the target 7003. When the user moves the pointer to a border 
of the Media Warehouse window, the contents of the Media Warehouse 
window scroll without releasing the mouse button. The cursor may be 

15 changed when a media element 1 690 is selected as a target 7003. 

Depending on the type of target 7003 selected, a Target Properties Sheet 
7703, described below, may pop up when the mouse button is released. 
The user may then enter the properties of the target 7003 into the Target 
Properties Sheet 7703. Other techniques to link 1010 a hotspot 8003 to a 

20 target 7003 may be used. 



Target Properties and Multiple Targets 

Each hotspot 8003 can have multiple targets 7003. Thus, while playing a 

hypervideo when the user actuates a mouse button when the pointer is over a 
25 hotspot 8003, one or more targets 7003 can be executed or activated. Multiple 

targets 7003 can be executed sequentially, in parallel or a combination thereof. 

When the user activates or actuates the hotspot 8003, muhiple targets 7003 can 

be executed in different levels 7601 (e.g., 1, 2, 3, etc.). Levels permit the 

multiple targets 7003 to be played simultaneously (same level) and sequentially 
30 (ascending levels). Each level has a Leader target 7602 which upon ending 

terminates all other targets 7003 in its level. 
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Alternatively, the targets 7003 can be positioned and displayed on a time 
line 7701, exemplified in Figure 7D. As a result, one target 7003 can be initiated 
during the performance of another target 7003. The time line 7701 can be 
illustrated on a story board page 7705 of the Target Properties Sheet 7703. 
5 When the user selects a target 7003 for a hotspot 8003, a Target 

Properties Sheet 7703, which may include a target dialog for the selected media 
element 1690, may be opened and a new target 7003 is created. The Target 
Properties Sheet 7703 may include a details page that lets the user enter target 
parameters. The Details page 7801, exemplified in Figure 7E, may include target 

1 0 7003 parameters that are unique to the selected media type and permit nonlinear 
authoring 1 009. In the Details page 780 1 , the user may choose whether the 
source media, such as video or bitmap, from which the target 7003 was launched, 
should Pause, Close or Keep Playing 7807 when the target 7003 is executed. 
The user may also specify how a target 7003 is terminated 7805. Options 

1 5 include: 

• Continue the hypervideo. 

• Go Back to the source from which the target originated, skipping 
subsequent level targets. 

• Loop - execute the target media in a loop. 
20 • Freeze the target's last frame. 

• Exit the hypervideo. 

A second, optional page in the Target Properties Sheet 7703 is a frame vydndow 
7901 , exemplified in Figure 7F, much similar to the preview page 6001 in the 
Media Properties Sheet 5003. The Target Properties Sheet 7703 includes, for 

25 some of the media types, a Range, or Frame, window that lets the user specify a 
segment of the target media that will be played. This option may be applicable to 
video and audio targets 101 1, 101 5. The frame window 7703, unlike the preview 
page 6001, lets the user define the IN and OUT frames and times 7903, 7905 for 
video and audio media files, respectively, to determine starting and stopping 

30 fi'ames and times for playing those media files. 

The Target Properties Sheet 7703 may also include a Display properties 
page 7950, for example, for visual media types, including, but not limited to, 
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video and bitmaps. In the Display properties page 7950, the user may choose 
how the target 7003 will be projected to the user, as described in the following 
tables. 

Bitmap targets 1013 may have a Place Properties page in their Target 
5 Properties Sheet 7703, exemplified in Figure 7G, that permits the user to position 
a Bitmap over a source element, such as a video. 

Upon entering target parameters into the Target Properties Sheet 7703, 
the user may press one of the following buttons: 

1 . OK 78 1 1 - Creates a target 7003 for a hotspot 8003 and folds the Target 
1 0 Properties Sheet 7703. 

2. Apply 78 1 3 - Creates a new target 7703 for a hotspot 8003 . Does not 
fold the Target Properties Sheet 7703. 

3. Cancel 7815 - Closes the last opened Target Properties Sheet 7703. 



1 5 corresponding parameters. The Sync parameter may not be permitted to be 
FALSE with any ending parameter setting other than Continue. 

Video Targets 



The following tables describe the different targets 7003 and 



The following table describes parameters for video targets 101 1 . 



Name 



Type 



Description 



PopWindow 



Move 



Check Box 



Check Box 



Specifies whether the video target 101 1 is 
executed, or played, in the main video 
window of the hypervideo project or pops 
up in a new window. TRUE means the 
target pops up in a new window. Default 
setting is FALSE. 

In case a pop-up window is selected, 
indicates whether the pop-up window 
should be moved when the user resizes the 
main video window. 



Resize 



Check Box 



In case a pop-up window is selected, 
indicates whether the pop-up window 
should be resized when the user resizes the 
main video window. 
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Name 



Type 



DiescriptioD 



Position 



RECT. 



Sync 



Check Box 



In' 



Out* 



Ending 



Button - 
Marker 

Buttons- 
Marker 

Combo Box 



When the video target 1011 plays in the 
pop-up window (Pop Window = TRUE), 
the user must enter the top left coordinates 
and the width and height of the new 
window. The RECT contains the format: 
left top width height. In case the width and 
height are absent, the default is the original 
size of the video. 

Specifies whether the hypervideo plays 
while the video target 1011 plays. This 
parameter is applicable only if the video 
target 1011 plays in the pop-up window 
(PopWindow = TRUE). The default 
setting is TRUE. When this parameter is 
FALSE, the Back option for ending the 
video is not applicable. 

This button places a videos marker which 
indicates the first frame to play in a range of 
frames in a video target 1011. 

This button places a videos marker which 
indicates the last fi-ame to play in.a range of 
frames in a video target 1011. 

Indicates what action to take when the 
video target 1011 ends playing. The 
options are: 

* Loop - Loops the video target 1011. 

* Back - Goes back to the source fi-om 
which the video target 101 1 originated. 
This option is not applicable in case the 
Sync parameter is set to FALSE. 

* Exit ~ Exits the hypervideo project 
player. 

* Freeze - Freezes the hypervideo. 

* Continue - Terminates the video target 
1011 and proceeds to the next target. 

The default option is Continue, 



' The In and Out options are supported by a slider 1 1,004 that controls the 
display of the video. The user moves the video to the desired frame and clicks 
the appropriate button. In case the In marker is placed after the Out marker, the 
authoring tool 1001 may ask the user whether the markers should be swapped or 
deleted. 
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Spatial-Temporal Relative Links 

Nonlinear authoring 1009 may also permit targets 7003, such as video 
targets 101 1, to be linked 1010 to temporal 1014, in addition to or in alternative 
to spatial 1012, coordinates of a hotspot 8003 in a source media. For a temporal 
5 relative link 1014, a base frame in a video target 101 1, for example, is defined for 
the first fi^e in which the hotspot 8003 appears in the source media element. In 
later frames of the source media element, the hotspot 8003 is linked 1010 to 
successive frames in the video target 1011 that are temporally related to the base 
frame. For a spatial relative link 1012, the position, and possibly the size, of the 
10 executed, or displayed, video target 101 1 is relative to the hotspot' s position or 
to the pointer's position when the hotspot 8003 is activated. 

Bitmap Targgte 

The following table describes some Bitmap target 1013 parameters. 

15 



Name 


Type 


Description 


Position 


RECT. 


The run-time module 1101 needs the top 
lefr coordinates and the width and height of 
the bitmap window. The RECT contains 
the format: left top width height. Incase 
width and height are absent, the default is 
the original size of the bitmap. 


Duration 


Milliseconds 


The duration in milliseconds for which to 
display the Bitmap target 1013. 


Sync 


Check Box 


Specifies whether the hypervideo should 
stop executing, or playing, until the bitmap 
target 1013 is no longer displayed. The 
default setting is TRUE. A FALSE setting 
does not allow the Back option to be set for 
the Ending parameter. 
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Name Type Description 

Ending Combo Box Indicates what to do when the bitmap 

target 1013 is no longer displayed. The 
options are: 

Back - Goes back to the source from which 
the video target 1011 originated. 
This option is not applicable in the 
case when the Sync parameter is set 
to FALSE. 

Exit - Exits the hypervideo project player. 

Continue - Terminates the Bitmap target 
1013 and proceeds to the next 
target. This is the only option 
that lets the user choose a new 
target after the bitmap is 
displayed. 



When a bitmap target 1013 appears or disappears from the display 
5 window, transition effects, including, but not limited to, the following, can be 
selected. The bitmap may grow from the middle or side of the display window, 
The bitmap may slide from the side of the display window. The bitmap may 
evolve from the hotspot. Finally, the bitmap may spirally grow on the display 
window. These effects can be implemented for videos also. 

10 



Md}Q Targets 



wo 99/10822 



PCT/US98/17444 



42 



Name 



Type 



Description 



Sync 
Mix 

Ending 



Check Box Specifies whether the hypervideo plays or 

pauses while the audio target 1015 plays. 

Check Box Indicates whether to mix the audio data 

with other data that is presently playing, or 
play the audio data '"on top*' of other data. 

Combo Box Indicates what to do when the audio target 
1015 ends playing. The options are: 

* Loop - Loops the audio target 1015 
back and forth. 

* Back - Goes back to the source from 
which the video target 101 1 originated 

* Exit - Exits the hypervideo project 
player. 

* Continue - Terminates the audio target 
1015 and proceeds to the next target. 
This is the only option that lets the user 
choose a new target after the audio 
target 1015 has played. 

The default is Continue. 



Audio targets 1015 can include the transition effects of fading in and out. 



Mixing of Several Audio Channels 

Audio tracks of several different media playing simultaneously may be 
mixed together. 
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Text Targets 
Name 



Type 



Description 



Position 



RECT. 



Font 
Size 

Duration 
Infinite 



Combo Box 

Combo Box 
Milliseconds 

Check Box 



Sync 



Check Box 



Ending 



Combo Box 



The run-time module 1101 needs the top 
left coordinates and the width and height of 
the bitmap window. The RECT contains 
the format: left top width height. In case 
width and height are absent, the default is 
the original size of the bitmap. 

The name of the font. Default may be 
Arial. 

The size of the text. 

The duration in milliseconds that the text 
target 1017 is displayed. 

Toggle between this parameter and the 
Duration parameter. When the user 
specifies the Infinite parameter, the message 
pops up as a message box and the user may 
actuate an OK button in order to exit this 
message box. 

Specifies whether the hypervideo should 
stop and wait for the duration of displaying 
the text to pass. The default is TRUE. In 
case the Infinite flag is ON, the Sync flag 
turns ON too. 

Indicates how to terminate the display of a 

text target 1017. The options are: 

Back - Goes back to the source from which 

the video target 1011 originated. 

Exit - exits the hypervideo project player. 

Continue - Terminates the text target and 

proceeds to the next target. This is the only 

option that lets the user choose a new 

target after the text target 1017 has been 

displayed. 

The default option is Continue. 



Text targets 1017 can include the transition effect of evolving letter by 

letter. 
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g^^ecutaWe Targgts 
Name Type 



Description 



Command Line Edit Box 



Position 



RECT. 



5 Sync 



Ending 



Check Box 



Combo Box 



Text describing the command line for the 
executable target 1 02 1 . 

The run-time module 1 101 needs the top left 
coordinates and the width and height of the 
executable window. The RECT contains the 
format : left top width height. 

Specifies whether the hypervideo should stop 
and wait for the duration of opening the 
executable target 1021 to pass. The default is 
TRUE. 

Indicates what to do when the executable 
target 1021 ends execution. The options are: 
Back - Goes back to the source fi-om which 
the video target 1011 originated. 
Exit - exits the hypervideo project player. 
Continue - Terminates the executable target 
1021 and proceeds to the next target. This is 
the only option that lets the user choose a 
new target after the executable target 1 02 1 is 
performed. 

The default is Continue. 



pditing OptiPffS 

1 0 The workshop window supports the following editing options: 

• Selection - different types are available: 

One hotspot 8003 in one frame; 
Multiple hotspots 8003 in one ft^ame; 

One hotspot 8003 over multiple frames (using the shift key, and the slider 
1 5 or arrow keys); and 

Multiple hotspots 8003 over multiple flames. 

• Undo and Redo - for: 

Deleting a hotspot 8003; 
Editing, such as cutting, copying and pasting; 
20 Defining a hotspot 8003 in more frames; 

Defining a new hotspot 8003; and 
Moving a hotspot 8003. 
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• Cut, Copy, Paste and Paste New - implemented to work for one or more 
selected hotspots for one or more frames. A user can cut and copy a 
selected hotspot and paste it on a different frame. Hotspots 8003 created 
with the paste command retain their original identity. However, hotspots 

5 8003 created with the Paste New command have new identities. 

• Delete and Delete Hotspot - will delete selected or entire instances of a 
hotspot 8003 in a media element 1690. Deletion can be performed on 
selected hotspots. When deleting all instances of a hotspot 8003, the 
hotspot will automatically be deleted throughout the media element 1690. 

10 When deleting selected instances of a hotspot 8003 in the authormg tool 

1001, the hotspot must remain selected. 



Attemate MQ^se Button 

By actuating the altemate mouse button, different sets of commands can 
15 be accessed. The type of command set accessed depends upon whether or not 
the pointer is over a hotspot 8003. When the pointer is over a hotspot 8003, the 
altemate mouse button may access the following conmiands: 

• Hotspot Properties 
Go To First Frame 

20 • Cut 

• Copy 
Paste 

• Delete One 
Delete Hotspot 8003 

25 • Bring To Front (see following Z-order of hotspots section) 

• Send To Back (see following Z-order of hotspots section) 

When the cursor is not over a hotspot 8003, actuating the altemate mouse 
button may access the following commands: 

• Workshop Properties 
30 • Snap /Stretch 

• Different sizes of the Workshop video 

• Preview 
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• Set In Frame 

• Set Out Frame 

• Go To In Frame 

• Go To Out Frame 

5 

Z-QrderofHotspots 

Hotspots 8003 should maintain a z-order among themselves when one 
hotspot 8003 overlaps another hotspot 8003. The user may place one hotspot 
8003 in back or in front of another hotspot 8003. Selecting a hotspot 8003 will 
10 not automatically bring it to the top of the z-orden This option may be supported 
by one or more of the following portions of the hypervideo environment 1000: 
the project file 1670, the authoring tool 1001, or the run-time module 1 101. 



Video Preview 

15 The Workshop window may provide the option to preview a hypervideo 

being edited in that window. 

The slider 6007 in the Workshop window permits the user to select and 
20 display one frame of a video. 



Floating Toolbars 

The Workshop Toolbar and Hotspot Toolbar may be floating. The 
toolbars can hook to the sides of the Workshop window. The orientation of the 
25 toolbars may be changed from horizontal to vertical. The toolbars may be 

displayed in a double vertical format. The interpolation buttons may be placed in 
the Workshop Toolbar. 

Authoring Tool GUI 

30 Figure 10 illustrates an exemplary graphical user interface (GUI) 10,001 

of the authoring tool 1001 . The authoring tool's GUI 10,001 includes the menu 
bar 10,005, the main tool bar 10,009, the Media Warehouse window 10,003, the 
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Workshop window 8001, and the Toois window 9001. A hypervideo 10,007 is 
shown in the Workshop window 8001 . 



2,15 Project View 

5 The project view 1007 illustrates the hypervideo story board 1116, shown 

in Figure 11, which illustrates the base target 1 1,001 and targets 7003 linked 
1010 to hotspots 8003. Icons 1 1,003 illustratmg a thumbnail illustration 4007 of 
hotspots 8003 in the targets 7003, 11, 001 are appended to the corresponding 
base target 1 1,001 and other targets 7003. Targets 7003 corresponding to the 
1 0 hotspots 8003 illustrated by the icons 1 1 ,003 may then be appended to the icons 
1 1 ,003 . The branch 1 1 ,005 of an icon 1 1 ,003 that has already been displayed in 
the story board 1116 may terminate. Branches 1 1 ,005 to targets 7003 of newly 
displayed icons are shown. In this way, the user can view the hotspots 8003 and 
targets 7003 of all media elements 1690 in the hypervideo project. 

15 

?,2 Rpp-T^me Mo<>wly 

The run-time module 1 101, illustrated in Figure IB, is used to play one or 
more hypervideos simultaneously. The run-time module 1 101 utilizes the data in 
the project file 1670. The run-time module 1 101 can be used to play hypervideos 

20 10,007 with different applications, including, but not limited to, Directors by 
Macromedia Incorporated (San Francisco California), Shockwave by 
Macromedia Incorporated, VDOLive by VDOnet Corporation (Palo Alto, 
Califomia), Netscape Navigator by Netscape Conununications Corporation 
(Mountain View, Califomia), and Internet Explorer by Microsoft. By 

25 communicating with the run-time library 1113 through player interfaces 1 1 02, the 
different applications use the run-time module 1 101 to play hypervideos 10,007. 
The player interfaces 1 102 include, for example: 

(1) plug-ins 1 1 03 for applications such as those described above (e.g., 
Directors and Netscape); 
30 (2) OLE control, such as ActiveX (OCX), 1 1 OS, for example, to 

integrate with Visual Basic environments by Microsoft Corporation; 
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(3) stand-alone modules 1 107 such as a stand-alone player 
incorporating the player interface 1 102; 

(4) an MCI driver 1 1 09 accessible by MCI; and 

(5) a software development kit (SDK) 1111. 

5 The run-time library 1113 permits the hypervideos 1 0,007 to be played. 

The run-time library 1113 may include four elements, exemplified below: 

(1) Execution of a hypervideo project by a project interpreter 1117. 
The project interpreter 1117 manipulates an object database 1 127. The object 
database 1 127 may include an object tree which will be subsequently described. 

10 Logging information and preparing statistics 1 147 pertaining to the hypervideo 
project execution can be recorded. 

(2) Administration including manipulation, for example, of media 
necessary for the execution of a hypervideo project is provided by a media 
manager 11 25, including an active media stack 1 141, a cache 1 143 of media that 

1 5 have been previously activated, and a look ahead system 1 145 containing media 
that are expected to be activated. The media manager 1 125 is used to manipulate 
media files, e.g., opening, playing and closing media files 1680. The media 
manager 1 125 reduces the time to open a file, by keeping track of the opened files 
and using the look-ahead system 1 145 to load new media files 1680 that may 

20 potentially be requested by the user. The newly loaded files are added to the 
object tree, subsequently described. 

(3) Manipulation of the hypervideo project, for example, including the 
object database 1 127, can be achieved through an interface 1 1 19 by activation 
feedback 1 129, and display and sound mechanisms 1131 Activation feedback 

25 1 1 29 permits control of the hypervideo project by user or programmatic actions 
1 149, 1 151 . Programmatic actions 1151 may be implemented in software, such 
as by the use of scripting languages 1004. A programmatic action 1151 may 
include the use of notification fi-ames 1 157, subsequently described. A user 
action 1 149 may include actuating a hotspot 8003, while a hypervideo project is 

30 being displayed, by, for example, actuating a mouse button, when the pointer is 
over the hotspot. As a result, a target 7003, such as a media element 1690, may 
be executed. 
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(4) Network communications capability 1115, for example on the 
Internet or an Intranet, that pemiits implementing the run-time module 1 101 in a 
client-server architecture 1121. The client-server architecture 1121 may permit 
the run-time module 1 101 to support multiple users 11 33. The client-server 
5 architecture 1121 may also permit the run-time module 1 101 to support 

streaming of video and other meta-information. A queue manager 1 137 manages 
a stream of information, including hypervideo information, being transmitted 
between the client and the server 1 121. 



10 2,21 Run-T ime Module Commands 

The run-time module 1101 may support the following commands: 

• Open project - Loads a specified hypervideo project (.OBV) file 1670. 
This operation may also modify the display window of the hypervideo 
project according to submitted window parameters. The window 

1 5 parameters include window size, position, z-order and hierarchical status. 

• Play project - Activates a previously loaded hypervideo project. A call- 
back object 12,001 may be supplied when using this command. 

• Stop project - Stops a playing hypervideo project. 

• Pause project - Pauses a playing hypervideo project. 

20 • Set Project Window Properties - Window properties of an open project 
may be altered. These properties include window size, position, z-order 
and hierarchical status. 

Add \ Remove A Call-Back Object - Call-back objects may be added or 
removed. 



25 



2.22 Run-Time Module Desig n 

An exemplary design of the run-time module 1 101 will now be described. 
This description will include an illustrative discussion of call-back objects, a 
player interface 1 102, and the run-time library 1113. 



30 



wo 99/10822 



PCT/US98/17444 



50 

Call-back Objects 

An application may include a call-back object 12,001, exemplified in 
Figure 12. Call-back objects are otherwise known as call-back windows or call- 
back functions. The call-back objects 12,001 couple the run-time library 1 1 13 to 
S an application 12,003, which may include the stand-alone module 1 107 or other 
applications described above. Specifically, the call-back objects 12,001 facilitate 
communications between the run-time library 1113, and the application 12,003. 
A call-back object 12,001 receives information, in the form of messages or 
function calls, upon the occurrence of certain events when a hypervideo 10,007 is 

1 0 performed. The call-back object 12,001 commimicates the information to the 
application 12,003. The events are listed in the following section describing 
Event Notification 1114. Furthermore, the application 12,003 can initiate actions 
in a hypervideo 10,007 through a call-back object 12,001 . The use of the term 
object in this paragraph and subsequently in section 2.2, refers, not to hotspots 

1 5 8003, but to objects in object oriented programs. 



gygnt NPtifipation 

The run-time library 1113 supports notification of call-back objects 
12,001 upon the occurrence of certain events (1 114). Upon such event 
20 notification 1114, call-back objects may present certain retum values to the run- 
time library 1 1 13, or initiating actions of the type listed below. Events include: 
• Hotspot related events - these events include: 

* Entrance of a hotspot 8003 into a fi-ame that is the first fi^me in 
which the hotspot 8003 is defined. 

25 * Exit of a hotspot 8003 from a frame that is the first frame in which 

the hotspot is not defined. 

* Activation of a hotspot 8003, when a user activates a mouse button 
when the pointer is over the hotspot 8003 in a certain frame. 

* Passing over a hotspot 8003, when the pointer is moved over a 
30 hotspot 8003. Hotspots may respond to this notification, for 

example, by specifying a cursor to be displayed. 
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Call-back objects 12,001 may respond to any one of these hotspot-related 

events. 

• Media Related Events - These events include: 

* Start or end of a media, when a media is activated or deactivated. 

5 The start and end of the media may not be the first and last frames of 

the media, but rather the first and last fiames of the media that are 
displayed. 

* Reaching certain marked frames or other previously marked points. 
Certain frames of a media may be marked during hypervideo project 

10 authoring, or during run-time, as notification frames 1 1 57. When 

notification frames 1 1 57 are executed or displayed, the run-time 
library 1113 notifies the call-back objects 12,001 . 

• Target Related Events - Specific targets may be marked for notification. 
Before activating, or executing, such targets 7003, call-back objects 

15 12,001 are notified. This notification allows the hypervideo 10,007 to 

abort the target 7003 execution or commence different related actions. 



Initiated Actions 

When a hypervideo 10,007 is playing, applications 12,003 which reference 
20 the hypervideo project may initiate two kinds of actions: 

1 . Actions that have a direct and apparent influence on the playing 
hypervideo 10,007. These actions include: 

Target 7003 Activation - whereby an application 12,003 can force 
the playing hypervideo 10,007 to launch, or execute, a certain 
25 target by creating a new target 7003 or by activating, or actuating, 

a hotspot 8003 linked 1010 to a previously defined target 7003. 

• Project Stop - whereby an application 12,003 can force a playing 
hypervideo 10,007 to stop. 

2. Actions directed at the project file 1670 that have an indirect influence on 
30 a playing hypervideo 10,007. These actions include: 

• Changing cursors - an application 1 2,003 may change the cursors 
associated with a particular hotspot. These cursors include 
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cursors that are used when a pointer is passed over the hotspot 
8003 or when a mouse button is actuated when the pointer is over 
the hotspot 8003. 

• Enabling or disabling hotspots 8003 - an application 12,003 can 
5 disable or enable a hotspot 8003. A hotspot 8003 can be fully or 

partially disabled. With full disabling, a defined hotspot 8003 is 
completely ignored. With partial disabling, specific hotspot 8003 
functions are disabled, such as predefined targets 7003, 
notification, and hotspot visualization 11 55, for example. 
1 0 • Replacing the base targets 1 1 ,00 1 - whereby an application 

12,003 can redefine the hypervideo base targets 1 1 ,001 which are 
played when the hypervideo 10,007 begins. This function may be 
invalid once hypervideo 10,007 has conmienced playing. 

• Marking notification frames 1 1 57 - whereby at run time, the 
15 application 12,003 marks frames for which notification is 

requested. 

• Querying the media and media position - whereby an application 
12,003 can retrieve at any given time the currently playing 
media(s) and its/their current position(s) (e.g., frame or time). 

20 

MCI Driver 

The run-time module 1 101 may include an MCI Driver 1 109 as one of the 
player interfaces 1 102 to permit the playing of hypervideos 10,007. The MCI 
driver 1 109 may allow Window applications to play hypervideos 10,007, using 

25 the Multimedia Control Interface (MCI) by Microsoft. An application 12,003, 
requesting to play a hypervideo 10,007, may send the appropriate MCI 
commands. The MCI may translate these commands into messages and send 
them to the MCI driver 1 1 09 corresponding to the file type that is being acted 
upon. The corresponding MCI driver 1 109 may recognize these messages and 

30 direct the run-time library 1113 appropriately. The MCI driver 1 1 09 may also 
process user input, for example, from a mouse or a keyboard. 
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In order to respond correctly to both sources of input, the MCI driver 
1 109 includes the following two components: 

(1) A driver procedure which responds to relevant messages sent by 
the system; and 

5 (2) A window procedure attached to a window. This procedure is 

designed to trap relevant mouse messages, and to serve as a call- 
back object 12,001 for notifications sent by the playing media 
elements 1690. 

In order to accomplish their tasks, the components use a set of hypervideo 
1 0 objects created and used by the run-time library 1113. Any time a target 7003 
has to be executed as a result of activation feedback 1 129, an object is 
constructed. These objects include information about a corresponding target 
7003. The objects that are playing or that were playing and may be played in the 
future, are kept in an object tree. In this object tree, objects that were activated 
15 from other objects are depicted as their descendants. Figure 13 illustrates 
exemplary objects 13,002 in an exemplary object tree 13,004. The objects 
13,002 and object tree 13,004 may also be used with player interfaces 1 102, 
including the MCI driver 1 109, described above. The MCI driver and window 
procedures will now be described in more detail to exemplify the use of objects 
20 1 3,002 and the object tree 1 3,004. 

MCI PrivCT PrQggdwe 

The MCI driver 1 1 09 may be designed to respond to the messages sent to 
it by the system. These messages may include general driver messages, and MCI- 

25 specific messages. The MCI messages are interpretations of the messages sent by 
an application. The messages handled by the driver include: MCI_OPEN, 
MCI_PLAY, MCLSTOP, MCLPAUSE, MCLRESUME, MCLWINDOW, 
MCI_^CLOSE, MCI_PUT, and MCI^UPDATE. Custom messages may be 
developed to enable seeking specific clips in given frames or targets 7003. An 

30 exemplary description of some of the messages follows: 

MCI_OPEN (relevant parameters: Open-Flags, Open-Parms): 
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This message initiates the hypervideo 10,007 according to the file name 
found in the Open-Parms. When processing this message, the base target 
11, 001 to be played becomes the root object 13,006 of the object tree 
13,004. According to the Open-Flags and Open-Parms, a window is 
5 created. The window's procedure is set to be the driver's WndProc. 



MCLPLAY: 

This message starts playing the hypervideo 10,007. The message can be 
received, for example, upon one of the following situations arising: (1) 

1 0 The project is stopped - In this case the object tree 1 3,004 is initiated 

from the root object 13,006. The base target 1 1,001, corresponding to 
the root target 13,006 is thus played. (2) The project is paused - In this 
case all the objects 13,002, or corresponding targets 7003, which have the 
status flag Playing (e.g., were previously playing, see MCI_PAUSE) are 

1 5 played. 



MCLSTOP: 

The hypervideo 10,007 and all of its playing targets 7003 are stopped, and 
the object tree 13,004 is collapsed. 

20 

MCLPAUSE: 

The hypervideo 10,007 and all of its playing targets are paused. Pausing 
does not affect the status flag of the objects 13,002. Thus, when 
resimiing the playing of the hypervideo 10,007, the run-time module 1101 
25 knows which objects 1 3,002 to resume playing. There is no effect on the 

object tree 13,004. 



MCLRESUME: 

This message is effective only if the hypervideo 10,007 is paused. In this 
30 case all the objects 13,002 of the hypervideo 10,007 marked Playing are 

played. 
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MCI_WINDOW (relevant parameters: window): 

This message replaces the window relating to the •fiill-screen* of the MCI 
driver 1 109. It can be sent only when the MCI driver 1 109 is not playing. 
The window is attached to the root object 13,006, and is subclassed to 
5 WndProc. 

MCLCLOSE: 

This message releases all memory, and terminates the MCI driver 1 109. 

10 Window Procedure 

Each driver session is associated with a window. The association is. made 
by means of the MCI_OPEN and MCI_WINDOW commands. The association 
must be done before playing starts. The driver uses the window procedure to 
obtain mouse and MCI events. In case the driver is assigned a new window using 

1 5 the MCI__WINDOW command, it will subclass its window procedure. Any other 
child windows which are created during the playing of the hypervideo 10,007 are 
also subclassed to this same procedure, and the interaction with them is done 
through it. 

The main messages handled by the window procedure are: 

20 

WM_LBUTTONDOWN (relevant parameters: window, mouse position): 

This message is used to identify user action of actuating a mouse button. 
From the window handle, the object 13,002 can be retrieved via the 
management manager 11 25. After identifying the object 13,002 and the 
25 pointer coordinates, the object database 1 127 is queried for a target 7003, 

or object 1 3,002, to be executed. According to the data received from 
the object database 1 127, the current object 13,002 may be stopped. 

WM^SETCURSOR (relevant parameters: window): 
30 This message is used to set the cursor according to the pointer 

coordinates, whether the pointer is over a hotspot 8003, whether a mouse 
button has been actuated, and depending upon the media element 1690 
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being executed. From the window handle, the object 1 3,002 can be 
retrieved via the management system. 



MM_MCINOTIFY (relevant parameters: device ID): 
5 This message is used to signal that an object 13,002 has terminated. From 

the device ID, the object 13,002 is retrieved via the media management 
system 1 125. Notification of medias that terminated unsuccessfully may 
be ignored at this stage. The object 1 3,002 that ended is updated to show 
that the corresponding media file 1680 is no longer playing. For 
1 0 compound objects, this may not necessarily mean that the object 1 3,002 

has terminated. If the object 1 3,002 is terminated, then a check is 
performed on the object's termination flag, or ending parameter, 
previously described. The following actions are taken according to the 
following flag settings: 
1 5 if EXIT stop all playing target(s) 7003 . 

if LOOP play target(s) 7003 again. 

if CONTINUE stop target(s) 7003 and take it/them off the active 
media stack 1141. Get the next object 13,002 and play it. 
if BACK stop target(s) 7003 and take it/them off the active media 
20 stack 1141. Continue the parent obj ect or target. 

if FREEZE pause target 7003. 



MM^MCISIGNAL (relevant parameters: device ID, video position): 

This message is used to track a playing object 1 3,002. Again, the object 
25 1 3,002 may be retrieved from the device ID. Tracking a playing object 

13,002 permits the following: 

1 . Since the image in a video, for example, may change, the cursor 
may have to be altered even though the mouse has not moved. 
For that reason, it may not be sulficient to wait for the 

30 WM_SETCURSOR message. 

2. Identiiying notification frames 1 1 57. For each fi^e, the object 
database 1 127 is queried to determine if the fi^me is a notification 
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frame 1 157. If the frame is a notification frame 1 157, the 

corresponding target(s) are executed. 
3. Logging, for example in a database, the entrance, or appearance, 

and exit, or disappearance, of hotspots 8003. 
5 An object 13,002 always keeps its position, for example frame nimiber, of 

its last signal message. When such a signal message is received, the object 
13,002 is queried by itself for changes that occurred between the last kept 
and current position. 



1 0 WM_TIMER (relevant parameters: device) 

This message is used to signal that a timed object has terminated. The 
action is similar to the notify message. 

Objects and the Object Tree 

15 The object 13,002 includes a description of a corresponding target 7003. 

Additionally, the object also may include status information, such as playing 
status, the MCI device ID and relevant window. 

The objects 13,002, or corresponding targets 7003, that are being 
executed, or that were executed and still may at anytime in the future be executed 

20 may be kept in an object tree 1 3,004 The objects 1 3,002 of the object tree 
13,004 may be of two general types: visible 13,008 or invisible 13,010. The 
visible objects 13,008, unlike invisible objects 13,010, can serve as parents of 
other objects 13,002. Invisible objects 13,010 are always leaves. 

An object 13,002 may be added to the object tree 13,004 once the object 

25 1 3,002 starts playing. The object 1 3,002 is removed from the object tree 1 3,004 
once the object 13,002 is no longer scheduled to be played. An object 13,002 
may be added to the object tree 13,004 in the follov^ng situations: 

1 . When starting to play the hypervideo 1 0,007, a root object 1 3,006, 
representing the base target(s) 1 1,001, is created and is placed at the top 

30 of the object tree 1 3,004, 

2. As a result of actuating a switch, or button, on a mouse or reaching a 
notification frame 1 157, an object 13,002 is spawned from the object 
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13,002 that contains, for example, the hotspot 8003 over which the 
pointer was placed when the switch was actuated, or clicked. 
3. If an object 13,002 ends or is closed, and its termination flag is set to 

continue, the closed object 13,012 is destroyed and a new object 13,014 is 
5 placed as a son of the closed object's parent object. 

An object 13,002 may be removed from the object tree 13,004 in the 
following situations: 

a. When the hypervideo 10,007 reaches its end or the close 
command is received, the object tree 13,004 is terminated. 
10 b. If an object 1 3,002 terminates and the termination flag is set to 

continue (used to activate new object 13,014) or to back (used for 
asynchronous objects to continue playing their parent object ). 
Objects 13,002 removed from object tree 13,004 may be marked in one of 
two ways: (1) stop all the playing descendant object(s) or (2) do not stop all 
15 playing descendant object(s). If an object 13,002 of the first type (1) is removed 
from the object tree 13,004, then descendent object(s) that branch from the 
removed object are also removed from the object tree 13,004. If the object 
13,002 of the second type (2), is removed from the object tree 13,004, then 
branch(es) of descendent object(s) of the removed object are appended to the 
20 parent object of the removed object. 

Mpdii^ Mmm^T 

The media manager 1 125 includes a list of media file 1 680 names. The 

media manager 1 125 also includes an MCI device ID for each media file 1680 in 
25 the list. The media manager 1 125 fiirther includes a list of objects 13,002 that 

use those device IDs. As a result, the objects 13,002 can interact with the MCI. 
The media manager 1 125 supports the following commands: 
Open (media file, object reference, window) - this command opens a 

media file 1680 in a specified window and returns the MCI device ID of the 
30 opened media file 1 680. If a copy of the specified media file 1 680 has previously 

been opened and is available, the same media file 1680 copy and MCI device ID 

will be used. 
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Stop (object reference) - stops the playing of the media file 1680 relating 
to the object reference. 

Close (object reference) - removes the object. 

5 Patabage Resolution 

Resolution of a hypervideo project may be 1000 by 1000. To optimize 
the speed of database calculations, the resolution may be 1024 by 1024. Also, 
the user can change the resolution of the hypervideo project coordinates. 

10 Pointer Location 

The Microsoft Foimdation Class (MFC) CRgn objects may be used to 
determine whether the pointer is positioned over a polygon-shaped hotspot. 

Overlaying Displayed Media Files 
15 Two or more media files 1680 may be displayed overlapping each other. 

A z-order relationship is maintained between the media files 1680 so that they are 
displayed over one another in a consistent fashion. Special overlay techniques, 
such as key color, blue screen and alpha channel may be used to achieve sprite 
animation and non-rectangular overlaps of windows. 

20 

HptspQt VistiaUzatjon 

A marker 1 155 can be attached to a hotspot 8003 (i.e., the hotspot 8003 
can be marked). The marker 1 155 can serve two purposes: First, the marker 
1 155 signals the user that a hotspot 8003 exists. Second, the marker 1 155 can 
25 signify that the user has already actuated the hotspot 8003. 

To facilitate these purposes, the marker 1 155 can be attached to the 
hotspot 8003 in one of the following ways: 

• Statically by showing the marker continuously. The marker 1 155 can be 
turned on or off by an application 12,003, such as a monitoring 
30 application; or 
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Dynamically when, for example, the hotspot 8003 is actuated by an 
application 12,003, such as a monitoring application, or when a pointer is 
over the hotspot 8003. 

The marker 1 155 may be implemented using one of the following 
5 techniques: 

(a) A text tag attached to the hotspot 8003 ; 

(b) A bitmap tag attached to the hotspot 8003; 

(c) Overlapping the hotspot 8003 with a surface, such as a translucent 
surface to highlight the hotspot 8003; 

1 0 (d) With sprite animation; and 

(e) With tool tips. When a mouse moves over a hotspot 8003, a tool 
tip may pop up to specify, for example, the hotspot' s name. 



Run-time Statistics 

15 Statistics 1 147 are gathered for hotspots and media files 1680. Statistics 

1 147, for example, include the nimiber of times a hotspot 8003 has been activated 
or the number of times a particular media file 1 680 was played. The statistics 
1 147 can be accessed used by applications 12,003, such as monitoring 
applications, to initiate conditioned actions. The statistics mechanism can be 

20 replaced, at least partially, by a counting mechanism in the application 12,003 that 
is triggered by event notification 1 1 14. Additionally, statistics 1 147 can be 
displayed. 

Proiect Status 

25 The current status of a project, including currently playing media, for 

example, defined by the object tree 13,004 and those waiting in the active media 
stack 1 141 , can be saved by the run-time library 1113. The saved status can later 
be recalled to continue playing a project from a specific point. 
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2.,^ Streaming Hypervideo 

Hypervideo data can be streamed, as described above, for example across 
a network 1650 from a server to a client, as described above. Because the 
5 hypervideo data is streamed, instantaneous network bandwidth consumption is 
diminished. Also, latency, associated with hypervideo performance, may be 
reduced. Techniques for streaming hypervideo data will now be further 
described. 

10 Media Objects 

In one embodiment, media objects may be hotspots 8003 in visual media, 
such as a videos and bitmaps, that may or may not be associated with interactive 
features, such as links to targets 7002. In another embodiment, media objects, 
however, may be associated with other hypervideo features such as targets 7003 

1 5 (or links), cursors (e.g. click and over cursors), and markers 1155. 

Media objects may be referenced in an object oriented manner, in addition 
to a traditional image oriented fashion, such as by reference to a frame or time 
base. Because media objects may be referenced in an object oriented manner, 
media objects may be referenced in a relatively natural manner. Not only is 

20 interactivity permitted with media objects, but so also is storage and retrieval of 
media objects (or object based indexing of media), and media object statistics. 

Streaming of Medi a Objects 

In one embodiment, media object data may be included in streaming 
25 multimedia. In another embodiment, media object data may be included in the 
hypervideo data stream 1660. 

In a further embodiment, the multimedia stream may include a video and 
audio stream. In yet another embodiment, the video stream may include a variet>' 
of visual information including an image, a slide show, and/or an animation. 
30 The hypervideo data stream 1 660 may permit a user to interact with a 

video stream in a manner similar to that described above. In one embodiment, the 
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hypervideo data stream 1660 may include information, such as geometry and 
shape, about a media object in a frame of a video. 

Hypervideo data may be extracted from the hypervideo data stream 1660, 
and may be provided to a hypervideo application, with an application program 

5 interface (API), as will be further described below. Streaming hypervideo may be 
displayed and accessed by a user by modifying the run-time module 1001 to 
include a hypervideo data stream 1660 renderer. The hypervideo data stream 
1660 renderer may include an API that provides access to media object 
information in the hypervideo data stream 1660, and may permit complex 

1 0 interaction with media objects. 

To diminish latency and instantaneous network bandwidth consumption, 
the hypervideo data may be streamed with the streaming multimedia. The 
hypervideo data may, however, be dovmloaded prior to the streaming of the 
multimedia. However, this downloading approach may undesirably increase 

1 5 latency and instantaneous network bandwidth consimiption which may scale, for 
example linearly, with the length of the streaming media. 

Streaming hypervideo data may also be desirable to accommodate the 
dynamic nature of a video. Video may be dynamic in nature. For example, the 
geometry of the media object may change from frame to frame, for example, due 

20 to variations in perspective. If, however, the geometry of the media object in the 
hypervideo data stream 1660 is fixed over a range of frames, the media object 
geometry may not be accurately defined in the range of frames. As a result, user 
interactivity with the hypervideo 10,007 may be undesirably diminished. 
Therefore, in order to enhance user interactivity, media object data in the 

25 hypervideo stream may be dynamically provided, for example for each frame in a 
corresponding video. One technique for streaming hypervideo data for each frame 
of a corresponding video will now be described. 

Streaming Hvpervideo Data F or Each Video Frame 
30 The following technique may implement streaming hypervideo by 

providing hypervideo data for each frame of a corresponding video. This 
technique has the advantage that the hypervideo data stream may be randomly 
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accessed at any time in the stream. Further, this technique may used with no, or a 
relatively small, hypervideo data stream header so that less memory is required of 
a recipient client to store such a header. 

Further, this technique may permit manipulation of hypervideo data 
5 streams 1 660, and exportation of hypervideo data into a hypervideo data stream 
1660 of a streaming mukimedia file. In one embodiment, the streaming 
multimedia file may be an audio video interlace (AVI) file. A hypervideo data 
stream 1660 may be stored in a private track of an AVI file. This embodiment 
may be implemented with computer software, such as file Hvstrvw.cpp, 

1 0 illustrated below. The operation of the computer software will now be described. 

A class ChvstreamView may include a method OnStreamAddstream. In 
one embodiment, when invoked from an object instantiated fi'om class 
ChvstreamView, method OnStreamAddstream may export hypervideo data from 
a project file 1670 through an authoring tool 1001 to a private track of a 

1 5 designated AVI file. This exportation process may be repeated for each AVI file 
in a hypervideo 10,007. 

The method OnStreamAddstream will now be described. A designated 
AVI file may be opened for reading and writing. Then, the streaming video track 
in the AVI file may be accessed. Information from the streaming video track may 

20 be read. This information may be used, in part, to create the hypervideo data 
stream 1660 stored in the private track of the AVI file. In one embodiment, the 
information fi-om the streaming video track may be used to create the header 
1402 and body 1404, illustrated in Figure 14, of the hypervideo data stream 1660. 
The body 1404 may be adjacent, or otherwise coupled, to the header 1402. 

25 The body 1404 of the hypervideo data stream, which may be stored in the 

private track, may be created in the following manner. The hypervideo data (e.g. 
hotspot data) for each frame in the AVI file may be extracted and stored in the 
private track. For example, every hotspot 8003 associated with, or in, each frame 
of the AVI file may be identified, and corresponding hotspot 8003 data may be 

30 extracted. The hotspot 8003 data corresponding to each fi-ame may be stored, in 
the body 1404 of the private track, on a frame by frame basis. The hotspot data 
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may be stored in an order corresponding with the order of the video frames. 
Then, the AVI file may be closed. All AVI files in the hypervideo 10,007 may be 
so modified. In one embodiment, this and other techniques for creating files with 
streaming hypervideo data may be implemented with modified authoring tools, 
5 fiirther described below. 

Because hypervideo data is provided for each frame of a corresponding 
video, this technique may consume a relatively large amount of network capacity. 
Thus, this technique may be relatively costly. Therefore, it may be preferable to 
use a more efficient technique for streaming hypervideo data. 

10 

V^ ypervideo Data Stream with a Header and a Body 

Network channel capacity may be reduced by placing static hypervideo 

1 5 data in the header 1 402 of the hypervideo data stream 1 606. For example, static 
hotspot data may be placed in the header 1402 because the static hotspot data 
does not vary throughout the hypervideo 10,007. Because it is placed in the 
header 1402, static hypervideo data may be transmitted once, unlike in the 
technique described above, reducing transmission bandwidth consumption. 

20 Static hotspot data may include data about the click and over cursors, a 

marker 1 155 and targel(s) 7003 defined for a hotspot 8003 because such data 
may remain constant throughout the life-time of the hotspot 8003. Specifying 
such attributes more than once for each hotspot 8003 (e.g. for each frame in 
which the hotspot 8003 occurs) may be undesirably redundant as described 

25 above. 

This streaming technique, like the previously described technique, 
facilitates the exportation of hypervideo data into a hypervideo data stream 1660 
of a streaming multimedia file. In one embodiment, the streaming muhimedia file 
is an audio video interlace (AVI) file. The hypervideo data stream 1660 may be 
30 stored in a private track of the AVI file. In one embodiment, this embodiment 
may be implemented with computer software, e.g. file ExportAvi.cpp illustrated 
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below. The operation of the computer software will now be described. 

In one embodiment, file ExportAvi.cpp includes the following arguments: 

a_sProjectFile which is the name of the hypervideo project file 1670 
which includes media file(s); 
5 a_sMediaName which is the name of the AVI file that includes the audio 

and video streams to be incorporated into a new AVI file; and 

a_sTargetAVI which is the name of the new AVI file in which the 
streaming hypervideo data 1660 will be written on a private track. 

The computer software operates in the following manner. A project file 
1 0 1 670 may be opened. In one embodiment, the data in the project file 1 670 may 
be encrypted. 

Hypervideo data may be extracted from the project file 1 670 identified by 
the a_sProjectFile argument. In one embodiment, the extracted hypervideo data 
may be stored in an EProject object. An EProject object may be an object 
1 5 oriented representation of the database in which hypervideo data is stored, for 
example in the authoring tool 1001 . The EProject object is further described 
below. 

The hypervideo data may be extracted fi-om the project file 1670 in the 
following manner. The amount, or length, of data in the project file 1670 may be 

20 determined. A buffer, having such length, may be allocated. Encrypted data is 
read from the project file 1670, and may be stored in the buffer. The encrypted 
data stored in the buffer may be decrypted. A memory file may be created, and 
attached to the buffer. Memory files and buffers are well known to persons 
skilled in the art. An archive may be attached to the memory file. The data in the 

25 buffer may be transferred to the EProject object. The archive and project file 
1 670 may be closed. 

Assuming that the extraction process is successfully completed, 
hypervideo data associated with the selected AVI file, in the hypervideo and 
identified by the a_sMediaName argument, may be transferred to a new, 

30 corresponding AVI file, identified by the a_sTargetAVI argument. The 

hypervideo data may be written in the private track of the new AVI file. The 
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hypervideo data may be written in the private track of the new AVI file. The 
hypervideo data transfer may be performed in the following manner. 

Streamed hypervideo data may be extracted from the EMedia object that 
corresponds to the selected AVI file. An EMedia object may represent, in an 

5 object oriented manner, the AVI file and its associated hypervideo data. The 
EMedia object corresponding to the selected AVI file may be selected in the 
following manner. EMedia objects may be iteratively analyzed to uncover the 
EMedia object that is associated with the selected AVI file. 

Upon ascertaining the EMedia object corresponding to the selected AVI 

1 0 file, the new AVI file may be prepared. If the new AVI file does not exist, then 
the new AVI file may be created, including by copying the streaming audio and 
video data from, the selected AVI file to the new AVI file. If the new AVI file 
already exists, but does not include the same video stream as the selected AVI 
file, then the selected AVI file's audio and video streams may be copied to the 

15 new AVI file. 

The file ExportAvi.cpp includes method AddObjectlnformation which 
creates a hypervideo data stream 1660 in the private track of the new AVI file. 
This function includes the arguments from the EProject object (corresponding to 
the selected project file 1 670) and the EMedia object (corresponding to the 

20 uncovered EMedia object, and including hypervideo data and pointer to the 

selected AVI file), and the name of the new AVI file. The method may operate in 
the following manner. 

Video For Windows (TM) software library may be initialized. A pointer, 
in the uncovered EMedia object, to an EClip object may be acquired. The class 

25 EClip, derived from the class EMedia, may include parameters for storing 

corresponding hypervideo data. In one embodiment, the classes EClipObject and 
EClipFrame, derived from class EClip, may include parameters for respectively 
storing static hotspot data (e.g. target, cursor, marking, and frame range data) 
and dynamic hotspot data (e.g. geometry and shape data). 

30 In one embodiment, the static hotspot data may be acquired in the 

following manner. The number of fi-ames in the video stream of the selected AVI 
file may be determined. Static hotspot data may be extracted for each hotspot by 
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indexing the EClipObject objects associated with the ECIip object coiresponding 
to the selected AVI file. The extracted static hotspot data for each EClipObject 
object may include data about a click cursor (1 byte), an over cursor (1 byte), a 
Target URL (1+MAX_URL byte), and a hotspot object frame range (2 DWORD 

5 =2*4 bytes). In other embodiments, the static and dynamic hotspot data may 
include parameter data different than described in this section. The static hotspot 
data may be stored in the allocated memory to form the header 1 402 of the 
hypervideo data stream 1660 in the private track of the AVI file. In one 
embodiment, each set of static hotspot data in the header 1402 may correspond 

10 to a unique instance of a hotspot object. Other data obtained from the video 
stream of the selected AVI file may also be used to form the header 1402. 

In another embodiment, the dynamic hotspot data (e.g. hotspot geometry, 
shape) may be similarly acquired by indexing the EClipFrame objects in the EClip 
object corresponding to the AVI file. The dynamic hotspot data may be 

1 5 reorganized by firame, rather than by hotspot, and copied to the allocated memory 
to form the body 1404 of the hypervideo stream data in the private track of the 
AVI file. In one embodiment, dynamic hotspot data may be provided for each 
video frame. The dynamic hotspot data may be stored in the allocated memory to 
form the body 1404 of the hypervideo data stream in the private track of the AVI 

20 file. 

If a frame of video is identified as key frame in the video stream of the 
selected AVI file, the corresponding hotspot data associated with the key frame is 
identified as key hotspot data in the hypervideo data stream 1660. Upon creating 
the hypervideo data stream 1660 in the private track of the AVI file, the AVI file 
25 is closed. 

H ypprviHen Data Stream Having D ynamic HotsDOl Information Provided fpr 
Multiple Video Frames 

A further technique for diminishing network capacity consumption will 
30 now be described. This technique permits the hypervideo data rate to be reduced 
below the corresponding video frame rate. Unlike the previously described 
techniques, which required one set of streamed hypervideo data to be uniquely 
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associated for each video frame, this technique permits one set of streamed 
hypervideo data to be associated with one or more video frames. 

In one embodiment, when the set of streamed hypervideo data is received 
by the client in a client-server network, the geometry data, in the streamed 

5 hypervideo data, may be associated with each frame of the one or more video 
frames. In another embodiment, the geometry data associated with each frame of 
a set of more than one video frames may be determined by interpolating the 
geometry data of two consecutive sets of streamed hypervideo data sent 
respectively before and after the corresponding set of more than one video 

10 frames. 

Hypervideo data may be exported to a private (proprietary) stream in 
media file formats, including an ASF version 1 file, using hypervideo data buffers. 
Hypervideo data buffer streaming will be first discussed. Then, a technique for 
implementing hypervideo data in streamed hypervideo data buffers will be 
1 5 exemplified for a Netshow (TM) client-server system. 

pvpervideo Data B uffer Format 

In one embodiment, hypervideo data buffers may contain dynamic media 
object data about hotspots, including data about hotspot geometry, target(s) 
20 7003, and cursors 7003 . Hypervideo data buffers may also include media object 
data for notification frames 1 1 57, or hotframes, including target 7003 data. 

A hypervideo data buffer may include data about media objects associated 
with one or more video frames. When exporting a hypervideo 10,007 data to an 
ASF version 1 file, hypervideo data buffers may be created at user defined rates. 
25 The maximum hypervideo data buffer creation rate may be limited to a 

corresponding video's frame rate so that the maximum number of hypervideo 
data buffers equals the number of frames in the video. 

In one embodiment, hypervideo data buffers may be implemented with 
computer software, e.g. file NSCommands.cpp, illustrated below. File 
30 NSCommands.cpp may facilitate the creation of, and the extraction of hypervideo 
data from, hypervideo data buffers. Object oriented programming classes defined 
and implemented in file NSCommands.cpp may be 
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used in an authoring tool 1001 to create hypervideo data buffers. These and 
other classes may be defined in file NSCommands.cpp, for use in the run-time 
module 1001, to extract hypervideo data from hypervideo data buffers. File 
NSCommands.cpp includes the following three classes: 

5 ENSBuffer: Objects instantiated from this class read data from and write 

data to hypervideo data buffers. For example, this class's method 'Int' writes an 
integer to or reads an integer from a hypervideo data buffer depending upon the 
setting of 'storing' flag (e.g. if a hypervideo data buffer's 'storing' flag equals 
true the integer is written in the hypervideo data buffer when this method is 

10 performed). 

ENSFramelnfo: Objects instantiated from this class handle the creation of 
a hypervideo data buffer. 

ENSFramelnfoManager: Objects instantiated from this class manage the 
process of hypervideo data buffer creation. Upon exporting a hypervideo 10,007 

15 to a streaming media file, an object instantiated from the ENSFramelnfoManager 
class is called repeatedly according to the hypervideo data buffer creation rate 
every time a hypervideo data buffer is created. Each time an object instantiated 
from the ENSFramelnfoManager class is called, it receives an ENSFramelnfo 
object and an ENSBuffer object, and serializes the ENSFramelnfo object to the 

20 ENSBuffer object. 

RNSFramelnfo Class 

Objects instantiated fi^om the ENSFramelnfo class may include 
information about media object geometry and target(s) 7003 for a frame in the 

25 video. Objects instantiated from the ENSFramelnfo class may include an array of 
ENSHotspot objects, which includes location and target data for each hotspot in 
a frame, and an ENSEvent object which includes target data for the frame if the 
frame is a notification frame 1 157. Both ENSHotspot and ENSEvent classes 
inherit properties from a ENSFrameLinkData class which also includes 

30 parameters that describe target(s) 7003. A set of these parameters may be used 
to store Boolean variables for selecting target type. Hotspot 8003 and 
notification frame 1 1 57 target 7003 types may include: play another video, seek 
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to a different location in the same video, seek to an ASF Marker defined in the 
current video, display text on top of the video, or navigate to a location at a URL 
whose corresponding data is displayed in selected HTML frame. For each target 
type, the ENSFrameLinkData class includes a further set of parameters, or 
5 descriptors, used to further identify the target 7003. For example, if a target 
7003 type navigate to a location at a URL is selected, then the URL Boolean 
variable is set to true and the relevant descriptors, including the URL name, 
location and selected frame, are provided. The ENSHotspot class includes 
parameters to define hotspot 8003 geometry and cursors, in addition to the target 

1 0 7003 parameters inherited from ENSFrameLinkData. 

Hypervideo data buffers will now be further illustrated by exemplifying 
their use. A hypervideo data buffer may be created for a frame for which an 
elliptically shaped hotspot is defined. The elliptically shaped hotspot may be 
linked to a target 7003 which is of the type that displays text on top of the video. 

1 5 Thus, during hypervideo performance, upon actuating (e.g. clicking on) the 

elliptically shaped hotspot, text may appear over the video. The frame may also 
be a notification frame 1 1 57 whose target 7003 is the type that navigates to a 
location at a URL (e.g. for a web site) whose corresponding data is displayed in a 
selected HTML frame. The hotspot 8003 center is located at coordinates 

20 ( 1 00, 1 00) and the hotspot's horizontal and vertical radiuses are respectively 
(20,60). 

To implement the elliptically shaped hotspot and notification frame 1 157, 
an ENSFramelnfo object is instantiated. An ENSHotspot object is also 
instantiated. The ENSHotspot object may include an ENSEllipse object having a 
25 text type target. An ENSEvent object is created having a target 7003 type that 
navigates to a location at a URL whose corresponding data is displayed in a 
selected HTML frame. Both ENSHotspot and ENSEvent objects are added to 
the ENSFramelnfo object. An ENSBuffer object is then instantiated. The 
ENSFramelnfo object is serialized into the ENSBuffer object. 
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The resulting hypervideo data buffer has the following structure: 
Byte 0: t (stands for true, this is a notification frame) 

5 — the notification frame description 

Byte 1 : t (true, this notification frame has a URL type target) 

Byte 2: 22 (the length of the URL's name) 

Bytes 2-20: http://www.website.com (the URL's name) 

Byte 21 : f (false, this is not a seek to time target) 
1 0 Byte 22: f (false, this is not a seek to marker target) 

Byte 23: f (false, this is not a text target) 

— the hotspot description 

Byte 24: f (false, this is not a URl. target) 
1 5 Byte 25: f (false, this is not a seek to time target) 

Byte 26: f (false, this is not a seek to marker target) 

Byte 27: t (true, this is a text target) 

Byte 28: 8 (the text's length) 

Bytes 29-36: the text 
20 Bytes 37: 1 1 (hotspot' s description length) 

Bytes 38-48: Description 

Byte 49: 1 (the number of the hotspot click cursor) 
Byte 50: 1 (the number of the hotspot over cursor) 
Byte 51 : 100 (the ellipse center x-axis coordinate) 
25 Byte 52: 100 (the ellipse center y-axis coordinate) 
Byte 53: 20 (the ellipse horizontal radius) 
Byte 54: 60 (the ellipse vertical radius) 

R^heddinp the Buffers inside an ASF File 
30 This section describes how hypervideo data buffers may be placed in a 

streaming media file, such as an ASF version 1 file, which may include audio and 
video streams. As previously stated, the hypervideo data buffers may be inserted 
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into any media file that maintains a private data stream. In one embodiment, the 
hypervideo data buffers may be inserted into the script command stream in an 
ASF version 1 file. Script commands, used in the script command stream, have a 
type field, an argument field and an execution time field. In another embodiment, 

5 each hypervideo data buffer may be associated with a script command of type 
'VANS'. In another embodiment, the hypervideo data buffers may be streamed 
across a network 1650 from a server to a client including a Netshow (TM) player. 
The Netshow (TM) player may fire a script command, and hence the 
corresponding feature defined in the argument field, at the corresponding 

1 0 execution time. When a script command is fired, a corresponding media object 
data resident in the Netshow (TM) player may be modified. 

Additional Techniques to Reduce Network Cap acitv Consumption bv a 
^^YP^r^iden Data Stream 

1 5 The features of the techniques described above may be used to implement 

yet another technique for streaming hypervideo data in a manner that consumes 
less network capacity. For example, certain interactive hotspot 8003 attributes, 
such as click and over cursors, marker 1 1 55 and target(s) 7003, may be more 
likely to remain constant throughout the life-lime of the hotspot 8003. Specifying 

20 such attributes more than once for each hotspot 8003 {e.g. for each frame in 
which the hotspot 8003 occurs) may be undesirably redundant because such a 
technique consumes relatively large network capacity. Therefore, these static 
hotspot features may be defined in the header 1402 of the hypervideo data 
stream. On the other hand, however, providing such hotspot attributes, for 

25 example, for each video frame may be desirable because it may more readily 
permit streaming hypervideo to be randomly accessed, in mid-stream. 

Further, the content of successive video frames may be relatively time 
invariant (e.g. change little from frame to frame). In such cases, encoder- 
decoders (CODECS) may be used to compress video data to reduce the 

30 bandwidth of the video stream. Further, in such cases, media object geometry 
may also be compressed, for example with CODECs, to reduce the bandwidth of 
the hypervideo data stream 1660. In one embodiment, the CODECs may 



wo 99/10822 



PCT/US98/17444 



73 

compress the hypervideo data stream 1660 because the hypervideo data may be 
relatively time invariant also. In another embodiment, a CODEC may define a 
key frame in which media object information, including geometry, is encoded. 
Subsequent frames may be encoded based upon their differences, for example in 

5 hotspot 8003 geometry, with respect to the key frame. In another embodiment, 
which is relatively lossy, media object geometry, in subsequent frames may be 
defined by a vector, such as a motion vector. 

In another embodiment, redundant hypervideo data may be stored in a 
header 1402 of the hypervideo data stream 1660 that may be streamed to a run 

1 0 time module 1 1 01 in a client. In one embodiment, the redundant hypervideo data 
may be referenced by unique identifiers, or pointers, 1406 in the body 1404 of the 
hypervideo data stream 1660. 

The header 1402 may be stored in a buffer memory of sufficient size, for 
example in a client of a client-server network, during the display of the 

15 corresponding hypervideo 10,007. Further, because they may be loaded into the 
buffer memory, relatively large headers 1402 may increase latency, or download 
time, of the hypervideo data stream 1660. Latency may be more problematic for 
multimedia files that are randomly accessed after their beginning. In one 
embodiment, regardless of access location, a complete header 1402 may have to 

20 be loaded into the buffer memory before multimedia performance may begin. 

Hypervideo Data Streaming v ^th Common Data Pointers 

One embodiment of a technique of using pointers 1406, which may be 
referred to as the common data pointers method, will now be described. In this 

25 embodiment, typically no redundant, or common, media object data may be 
encoded in the hypervideo data stream 1660. The encoded stream may include 
only time-dependent data, such as media object geometry and shape. The 
common media object attributes, which are typically time invariant, and thus 
typically need only to be encoded once in the hypervideo data stream 1 660, may 

30 be stored in the header 1402 of the hypervideo data stream 1660. The common 
media object attributes are coupled to the media objects by pointers 1406. In one 
embodiment, the pointers 1406 are media object identifiers (IDs). 
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One embodiment of a hypervideo data stream 1660 will now be 
illustrated, and includes a description of the encoded parameters within the 
header 1402 and body 1404. 

5 ffea(ier: 

The header 1402 of the hypervideo data stream 1660 may include 
common attribute data associated with media objects encoded in the hypervideo 
data stream 1660. In one embodiment, the header 1402 may include the 
following common attribute parameters (primarily related to interactive features 
10 ofhotspots8003): 

Media Object ID: A unique identifier of a media object 8003. 
Cursors: User-defined cursors, including both the over and click cursors. 
Marking: Marker 1155 definition. 
Targets: Target 7003, or link, definition, 
15 Horizontal and Vertical Resolution, and Time Units, as described 

below. 

Dimensions of the Media Window in which the Media was Authored. 

The header 1402 will now be described in further detail. In one 
20 embodiment, the field for the Hotspot ID may be 32 bits in length. The cursor 
fields will now be described. 

In another embodiment, the field for the over cursor may be 2*8 bits. If 
the first byte is: 

0x0, then the over cursor is not changed fi"om the click cursor, and 
25 thus is the same as the click cursor; 

0x1 to Oxfe, then corresponding predefined cursor shapes are 

used; and 

Oxff, then the next byte holds a pointer to a location in a cursor 
table associated with a cursor shape to be displayed. 
30 In another embodiment, the field for the click cursor is 2*8 bits. The click cursor 
is defined in a manner similar to the definition of the over cursor. 
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In yet another embodiment, the field for the marker 1 1 55 is 4*8 bits. The 
type of marker 1 155 is described in 8 bits. If bit 1 is set to one, then a flag is set 
to indicate a blinking marker. If bit 2 is set to one, then the marker 11 55 is 
pointer sensitive, appearing when the pointer is positioned over the region 
5 corresponding to the media object. These techniques are further illustrated 
below. 

Body: 

In one embodiment, the body 1404 may include the following parameters 
1 0 for each media object: 

Media Object ID: Uniquely associated with a media object. 
Media Object Shape: Including Rectangle, Ellipse, Triangle, Polygon, 
Circle, etc. 

Media Object Geometry: Includes the size and position of the media 

1 5 object. The media object geometry may be encoded using relative coordinates 
with respect to the origin of the playback window to facilitate image map resizing 
to permit media object dimensions to be resized proportionally with the playback 
window. For media objects having primitive shapes, bounding rectangle 
coordinates may be provided, and include sufficient information to determine the 

20 geometry of the media object. Bounding rectangles, which are known to persons 
skilled in the art, are rectangles whose sides contact the periphery of a primitive 
shape, but do not intersect with the interior of the primitive shape. Bounding 
rectangles may be used to describe a triangle that is an isosceles triangle, for 
example, with a horizontal base. Other types of triangles, however, may be 

25 defined v^th a polygon shape. 

In one embodiment, the field for the media object shape has 8 bits. The 
data in this field identifies the shape of the media object (e.g. rectangle, triangle, 
ellipse, circle, polygon, etc.). For primitive shapes (e.g. rectangle, triangle, circle 
and ellipse), the media object dimensions are defined by a bounding rectangle. In 

30 one embodiment, the bounding rectangle coordinates, used to define a primitive 
shape media object, may be stored in a field of 4*16 bits. In another 
embodiment, only the top-left and bottom-right coordinates of the bounding 
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rectangle need be stored in this field. In one embodiment, for polygon shapes, the 
number of vertices are defined in a 16 bit field. The vertices' coordinates are 
defined in a field whose size equals the number of vertices 2*16 bits. 

In one embodiment, a common data parameter in the header 1402 may be 
5 associated with a media object parameter in the body 1404 by using an identifier. 
In another embodiment, the common data parameter for a media object may be 
used in the body 1404 by inserting the identifier (e.g. media object ID) 
corresponding to the common data parameter in a field, in the body 1404, 
associated with the media object parameter. 

10 

Scalable Streaming 

In one embodiment, latency and bandwidth of a hypervideo data stream 
1 660 can be diminished by streaming the hypervideo data in a scalable manner. 
The bandwidth of the hypervideo data stream 1660 may be varied to 

1 5 accommodate the bandwidth available in the system over which the hypervideo 
data stream 1 660 is transmitted. In this embodiment, substantial amounts of 
media object data may be encoded to form a relatively wide bandwidth 
hypervideo data stream. Alternatively, solely basic hypervideo data may be 
encoded to provide a relatively narrow bandwidth hypervideo data stream. One 

20 embodiment of a scalable hypervideo data stream will now be described. 

One embodiment of a scalable hypervideo data stream includes three 
layers of hypervideo data. In one embodiment, different layers of hypervideo data 
stream 1 660 describing one or more media objects are coordinated, utilizing 
object ID and time (or frame) parameters, in a manner understood by one skilled 

25 in the art. In another embodiment, the first layer of the hypervideo data stream 
1660 includes only basic hypervideo data, such as media object geometry, and no 
interactive data. The basic hypervideo data is intended to be utilized by any 
hypervideo application, such as one that utilizes indexing of media based upon 
media objects. A hypervideo data stream 1660 including only the basic 

30 hypervideo data (e.g. the first layer) consumes relatively little network capacity. 

In another embodiment, a second layer includes basic interactive features 
for the media object, including marker 1 155, cursor and target 7003 data. The 
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basic interactive features may be selected from predefined sets of markers 1 155 
(e.g. NoMarking, Highlight, Dark, Negative, Gray, RGB, etc.), cursors (e.g. 
NoCursor, HandCursor, etc.) and targets 7003 (e.g. URL flip [which changes the 
URL in a fi-ame of browser], Seek - a time code or frame. Switch stream, Pause, 
5 Exit, etc.). 

In another embodiment, the second layer may also provide other 
interactive features, such as markers 11 55 that alternate, including a blinking 
marker. The blinking marker is a marker that is turned on and off repetitively. 
The second layer may also enable the marker 1 155, cursors, and target(s) 7003 of 

1 0 a media object to change at different times or frames, for example, of a video in 
the multimedia stream. For example, a first instance of a media object may have a 
first marker, a first set of cursors, and a first target, and may be associated with a 
first time period, or set of frames, in a video. A second instance of the same 
media object may have a second marker, a second set of cursors, and a second 

1 5 target, and may be associated with a second time period, or set of frames, in the 
video. However, hypervideo data for a media object may be relatively time 
invariant, diminishing the number of instances of a media object, thus providing 
relatively high data compression rales and relatively low network capacity 
consumption. 

20 The third layer may include enhanced hypervideo properties, enabling 

user-defined markers, cursors and targets for hotspots 8003 to be defined in a 
hypervideo data stream 1660. Thus, such enhanced hypervideo properties may 
be used, without requiring customized software for implementing the 
enhancements, with the authoring tool 1001 and run-time module 1001. As a 

25 result, a user does not need to provide external code to utilize the enhanced 
hypervideo properties. 

Each of the three layers may provide a different scope of media object 
definition, including for geometry and interactivity. However, in other 
embodiments, the layers may be reorganized, for example, by merging or sub- 

30 dividing the layers. In a ftirther embodiment, for example, the first and second 
layers may be combined into a single layer to provide a non-scalable stream, for 
example, for a hypervideo system that does not support scalable streams. In one 



wo 99/10822 



PCT/US98/17444 



78 

embodiment, the single layer may include media object geometry, and cursor, 
marker 1 155, and target 7003 definition from the predefined sets discussed 
herein. 

An exemplary embodiment of the layers is described below: 



First lover: 

Media Object Shape: Rectangle, Ellipse, Triangle, Polygon, etc. 

Media Object Geometry: Indicates the size and position of the media 

object. For primitive shapes, two sets of 
1 0 coordinates of bounding rectangle may only be 

provided. The media object geometry may be 
encoded using relative coordinates to facilitate 
image map re-sizing, as fiirther described 
herein. 



15 Second Layer: 
Cursors: 



20 



25 



Marker: 



30 



Permits each media object to be assigned at least two cursors, 
the over and click cursors. The over cursor may be 
displayed, instead of the pointer, when the pointer is 
positioned over the region corresponding to the media object. 
The click cursor may be displayed instead of either the pointer 
or over cursor, when the media object has been actuated, for 
example by a mouse "click" when the pointer is over the 
region corresponding to the media object. In one 
embodiment, in the second layer, only cursors, such as 
NoCursor or the HandCursor, from a pre-defined set may be 
assigned to a media object. 

Permits each media object to be visually marked by a marker 
1 155. In one embodiment, in the second layer, only markers 
1 155, such as NoMarking, Highlight, Dark, Negative. Gray 
and RGB, from a pre-defined set may be assigned to a media 
object. 
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Target (Commands): Permits each media object to be coupled to target(s) 
7003. In one embodiment, in the second layer, only 
target(s), such as URL Flip, Seek, New Media, Pause, 
Resume and Exit, from a pre-defined set may be assigned 
5 to a media object. 

Third lover: 

Cursors: Permits user-defined cursors to be assigned to a media object 
by encoding the cursor's path in the hypervideo data stream 
1660. 

1 0 Marking: Permits a user-defined marker, such as an overlay image, or 

any low-bandwidth media, to be assigned to a media object by 
encoding the marker's 1 1 55 path and its display parameters 
(key color, relative position, etc.) in the hypervideo data 
stream 1660. 

15 Target(s) (Commands): Permits linking target(s) 7003, which may be any 

media, to a media object by specifying the target(s)'s location. 
Additionally, target display parameters, such as position, size, 
and duration, for example of a target window, may be 
provided in the third layer, 

20 

In one embodiment, if a media object parameters, such as cursor, marker 
1 155 or target(s) 7003, are defined, for example contemporaneously, in two 
layers (e.g. the second and third layers), the parameter definitions in the higher 
layer (e.g. third layer) may be used. If one or more parameters are not encoded 

25 in the higher layer (e.g. third layer), the parameters defined in the lower layer 
(e.g. second layer) may be used. 

As mentioned above, a hypervideo data stream 1 660 renderer may 
provide an extensive API to facilitate retrieval of streamed hypervideo data, such 
as media object information. For example, media object geometry may be 

30 extracted with the API, for example when only the first layer is streamed. 
Extended, or other, features may be implemented through the mentioned API, 
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Scalable Hvpervideo Da ta Stream Format 

Another embodiment of the hypervideo data stream 1 660 will now be 
described in further detail. 



5 Header; 

The header 1402 includes, in addition to other media parameters, global 
parameters required in the hypervideo data stream 1660. 



Field Name 



Field Type Size (bits) Description 



10 Horizontal Resolution UINT 



16 



The horizontal resolution 
of media objects. This 
parameter is used to scale 
the media object's 
geometry parameters, as 
described above. 



Vertical Resolution 



UINT 



16 



The horizontal resolution 
of media objects. This 
parameter is used to scale 
the media object's 
geometry parameters, as 
described above. 



Time Units 



OBTimeUn 8 



The time resolution, or 
units, of the visual media. 
This parameter defines the 
units used for time 
stamps. 



The Horizontal and Vertical Resolution parameters determine the 
1 5 resolution, or units, of media objects's geometry parameters. The Horizontal and 
Vertical Resolution parameters may be used to scale media objects's geometr\' 
parameters. The Horizontal and Vertical Resolution parameters recite a number 
of 'logical units' for frame v^dth and height. This relative representation readily 
facilitates re-sizing media objects and media scaling, or in other words allows 
20 media object size to be varied with the size of window in which the media object 
appears. 
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The Time Units parameter determines the time units with which the media 
object's time parameters, including start time, are defined. In one embodiment, 
the time units are defined in milliseconds (MS). In another embodiment, the time 
units are defined by fi*ame numbers. 

5 One embodiment of the body 1 404 of the hypervideo data stream 1 660 

will now be described. This embodiment is comprised of multiple layers. The 
structure of each layer, including the media object data therein encoded, or 
defined, will be described. The encoded media object data may include media 
object geometry and interactive features, as illustrated above. 

1 0 A media object may be defined for a finite period, or duration, of frames 

or time. In one embodiment, the finite period is valid from the Start Time of first 
data defining a first instance of the media object through the Start Time of second 
data defining a second instance of the media object. In another embodiment, the 
media object is terminated by issuing dummy data for the media object, including 

1 5 the media object's unique ID and End Time, which respectively specify the media 
object to be terminated, and the time to terminate the media object. 

In another embodiment, Time Stamps are provided, and can be used to 
define the beginning of a one set of media objects, and the end of another set of 
media objects. All media objects instantiated following the issuance of a time 

20 stamp are valid until the issuance of another Time Stamp. In this embodiment, all 
media objects share a common, and typically relatively fine, time resolution 
defined by the Time Unit parameter. 

Because this technique may use relatively finer time resolution than other 
techniques, this technique may be relatively inefficient for encoding media objects 

25 having relatively constant geometry for relatively long time periods. Media 

objects having relatively constant geometry for relatively long time periods must 
be repeatedly encoded with this technique. As a result, the hypervideo data 
stream 1660 bandwidth may be relatively large compared to other techniques. In 
the previously described technique, media objects can be encoded with a 

30 relatively wide time resolution. 

One embodiment of the data structures of the first, second and third layers 
is described below. 
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10 



First Lover 

Field Name Field Type 

Media Object ID UINT 

Start Time UINT 

Media Object Shape OBShape 

Media Object Flags OBFlags 

Media Object Geometry 



Size (bits) 
16 

32 



Description 

A unique identifier of the 
media object. 

The starting time of the 
instance of the media 
object. 

The shape of the instance 
of the media object. 

Different flags assigned to 
the media object that may 
be used by an extemal 
application. 



(4*16) or 
(N*2*16) 



For primitive shape media objects (such as a Rectangle, Triangle, Ellipse, 
Circle, etc) 



Left 



Top 



Right 



Bottom 



UINT 



UINT 



UINT 



UINT 



16 



16 



16 



16 



For Polygon shape media objects 
15 XI UINT 

UINT 



Yl 



Xn. 



Yn... 



UINT 



UINT 



16 



16 



16 



16 



X coordinate of the top 
left comer of the bounding 
rectangle. 

Y coordinate of the top 
left comer of the bounding 
rectangle. 

X coordinate of the 
bottom right comer of the 
bounding rectangle. 

Y coordinate of the 
bottom right comer of the 
bounding rectangle. 



X coordinator of the first 
vertex of the polygon. 

Y coordinate of the first 
vertex of the polygon. 

X coordinate of the n-th 
vertex of the polygon. 

Y coordinate of the n-th 
vertex of the polygon. 
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Field Name Field Type Size (bits) Description 

XN UINT 16 X coordinate of the last 

vertex of the polygon. 

YN UINT 16 Y coordinate of the last 

vertex of the polygon. 

The Media Object ID is a unique identifier of a media object throughout 
5 the life span of the media object. The media object ID is also used to reference, 
or associate, the media object extended parameters in higher (e.g. second and 
third) layers, with the geometrical parameters in the first layer. 

The Start Time parameter uses the units defined in the Time Units 
parameter in the header 1402, The Media Object Shape parameter is used to 
1 0 select one of a group of pre-defined shapes, including Rectangle, Triangle, 

Ellipse, Circle, Polygon, etc. The Media Object Geometry parameters are used to 
define the dimensions and position of the media object using the 
HorizontaWertical Resolution units defined in the header 1402. Primitive 
shapes, such as a Rectangle, Ellipse, Circle, Triangle, etc., are defined by a 
1 5 bounding rectangle. 



Rp.cond Layer 



Field Name Field Type Size (bits) Description 



Media Object ID 


UINT 


16 


A unique identifier of the 
media object. 


Start time 


UINT 


32 


The starting time of the 
instance of the media object. 


Over Cursor 


OBCursor 


4 


The shape of the cursor 
when moving over the media 
object. 


Click Cursor 


OBCursor 


4 


The shape of the cursor 
when clicking on the media 
object. 


Marking 




8 


The visual properties of the 
media object. 
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Field Name 



Field Type Size (bits) Description 



10 



15 



20 



Marking Type OBMark 



Blink 



Boolean 



Mouse Sensitive Boolean 



Link type 



OBLinkType 8 



Link Command: 

For URLCommand 

URL string 
Location string 

Frame string 

For Seek 
Commands 

MS/UNIT/ 

Time/Frame/Marker MARKER 

For New Media 
Commands 



N 
N 

N 



Media Path 



string 



32(64)/32/ 
32 



N 



The predefined marking to 
use. 

Determines whether marker 
blinks. 

Determines whether marker 
is mouse sensitive. 

The command type which 
will be activated when 
actuating the object. 



The full URL address. 

The location in the URL 
page to flip to. 

The frame within the browser 
frame-set to flip the URL in. 



The point in the stream to 
seek to. 



The full path of the new 
media to be played. 



25 



The media object ID is a unique identifier of the media object, throughout 
its life span. The media object ID is also used as a reference to the media object 
parameters in the higher (e.g. third) layers, and to the geometrical media object 
parameters in the first layer. 

The Start Time parameter is interpreted according to the Time Units 
parameter in the hypervideo data stream 1660 header. OBCursor defines one of a 
set of pre-defined cursors (e.g. NoCursor, Arrow, Cross, Hand, Back, etc.). 
OBMark defines one of a set of pre-defined markers 1 1 55 (e.g. NoMarking, 
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Highlight, Negative, Dark, Gray, RGB Meshes, RGB Filters, some pre-defined 
images marking such as an arrow or a flag pointing to the hotspot, etc.). 

If the Blink flag is on, or set to one, the selected marker will blink at a 
constant rate. If the Mouse Sensitive flag is on, or set to one, the selected marker 
5 will be displayed, for example in lieu of a pointer, only when the pointer is 
positioned over the media object. 

The Link Type defines the target 7003 linked to the media object. 
OBLinkType defines one of a set of pre-defined command (e.g. URL (flip a URL 
page), NewMedia (start playing a new media instead of the present media), 
10 SeekToFrame, SeekToTime, SeekToMarker, Pause, Resume and Exit). 



Third Lover 



Field Name 



Field Type Size (bits) Description 



15 



20 



Media Object ID 
Start Time 
Over Cursor 



Click Cursor 



UINT 



UINT 



string 



string 



Marking Media 

Marking Attributes 
Horizontal Anchor 



string 



16 



32 



N 



N 



N 



OBMrHAnc 2 



A unique identifier of a media 
object. 

The starting time of the media 
object. 

The full path of a cursor file. 
The over cursor will define 
the shape of the cursor 
displayed when the pointer is 
moved over the media object. 

The full path of the cursor 
file. This cursor will define 
the shape of the cursor when 
actuating, such as by clicking, 
the media object. 

The full path of the media 
used to mark the media 
object. 



Horizontal axis position of 
media object anchor. 



Vertical Anchor OBMrVAnc 2 



Vertical axis position of media 
obj ect anchor. 
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Field Name 


Field Type 


Size (bits) 


Description 


Transparency 


Boolean 




Use an opaque or transparent 
overlay. 


Blink 


Boolean 




Determines whether marker 
1155 blinks. 


Mouse sensitive 


Boolean 




Determines v^hether marker 
1 1 55 is be mouse sensitive. 


Not used 








iviaiiviiij^ iviwvijci 

Horizontal Offset 


UINT 


16 


The horizontal offset of the 
marker 1 155 from the media 
object anchor. 


Marking Media 
Vertical Offset 


UINT 


16 


The vertical offset of the 
marker 1155 from the media 
object anchor. 


Marker 

Transparency RGB 
Color (only if the 

llailo|JCU Wllwjr 1.1 Cl^ ID 

set or for text 
media). 


RGB 


24 


The background color of the 
marking media. 


Marking Media Blink 
Rate (only if the 
Blink Flag is set) 


UINT 


16 


The rate at which the marking 
blinks (in Time Units as 
defined in the stream header). 


The following Link Media, or Target, parameters can be repeated more than once 
for multiple Link Media to be support. 


Target Media 


string 


N 


The full path of the media 
which is linked as target 7003 
to the media object. 


Target Media 
Sequence Attributes 




16 




Link Media Level 


UINT 


8 


Level in which media object is 
located. The level may 
establish the order in which 
multiple targets are executed, 
as previously described above. 


Leader 


Boolean 


1 


Flag to indicate whether the 



target is a leader in a layer. 
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Field Name 



Field Type Size (bits) Description 



10 



Pause/ Close/ 
Keep Playing 
base media 

Pause/ Close/ 
Loop Target 

Not used 

Target Display 
Attributes 

Left 



Top 



Right 



Bottom 



UINT 



UINT 



UINT 



UINT 



3 

(4*16) 
16 

16 

16 



16 



What to do with the base 
media when the link media 
starts to play. 

What to do with the target 
when it ends. 



X coordinate of the top left 
comer of the bounding 
rectangle. 

Y coordinate of the top left 
comer of the bounding 
rectangle. 

X coordinate of the bottom 
right comer of the bounding 
rectangle. 

Y coordinate of the bottom 
right comer of the bounding 
rectangle. 



Any of the string parameters may contain a NULL if the parameters are 
1 5 not applicable, or if parameters in a lower layer parameters are to be used instead. 
If the Marker Media or the Target Media path string is set to NULL, all other 
corresponding parameters will not be encoded when authored or recognized 
when played. 

The Marker Media file may be any of the following types: Image file, 
20 Animation file, Low Bandwidth Video file or Text file. These media types can all 
be used lo mark a media object by overlaying the media relative to corresponding 
media object position. Note, the transparency attributes are defined by the 
transparency flag. 

To position the marker media relative to the hotspot 8003, an anchor is 
25 defined. In one embodiment, the anchor is a position in the hotspot, such as left, 
center, right or cxirsor position for the horizontal dimension, and the top, center, 
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bottom or cursor position for the vertical dimension. The exact position of the 
overlaying marker can be described by defining the hotspot anchor and the media 
offset from this hotspot anchor. If the marker media is text media, and the 
transparency flag is set to off, the transparency color will be used as a 
5 background color for the text media. 

The Target path can point to any file type, such as a media file, that is 
supported by a recipient client. In one embodiment, an relatively unlimited 
number of targets 7003 may be associated with each instance of a media object. 
The target(s) 7003 defined in the third layer supplement the target(s) 7003 

1 0 defined in the second layer.. 

The Target Media Sequence attributes define the sequence in which 
multiple targets 7003 are executed. Each target 7003 is assigned to a level 7601 . 
Targets on the same level, play simultaneously. Targets, on a layer such as the 
second and successive levels 7601, begin playing when the preceding level 7601 

1 5 is terminated, and the target(s) 7003 on the predecessor level 7601 have stopped 
playing, a level 7601 terminates when the target 7003 defined as the leader target 
7602 ends playing, as described above. The Target Media Sequence attributes 
also determine how base target(s) 1 1,001 are manipulated when the media object 
is actuated, and when each target 7003 ceases playing. The Target Media 

20 Display attributes define the position and size of the window in which a target 
7003 is displayed upon its execution. 

2.4 Streaming Hvp ervideo Authoring Tool 

Streaming hypervideo can be authored and displayed by modified versions 

25 of the authoring tool 1 001 and run-time module 1101. This section describes 
modified versions of authoring tools 1001 and run-time module 1101 for two 
streaming formats, Netshow (TM), by Microsoft (Redmond, WA), and 
RealVideo (TM), by Real Networks (Seattle, WA). The modified versions of the 
authoring tools 1001 and run-time module 1 101 are respectively derived from the 

30 authoring tool and run-time module 1101 described above. 

A modified run-time modules 1 101 plays streaming media fi-om Netshow 
(TM). a Real Networks (TM) player plays streaming media fi-om RealVideo 
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(TM) video servers. Netshow (TM) uses ASF. RealVideo (TM) uses a different 
video streaming format, the RM format. 

2.41 Hvpervideo Authoring Tool for Netshow fTM^ 
5 Modified versions of the hypervideo authoring tool 1001 and run-time 

module 1 101 for Netshow (TM) will now be described. The modified version of 
the authoring tool 1001 for Netshow (TM) can create Netshow (TM) compatible 
hypervideos. 

The modified version of the run-time module 1 1 01 for Netshow (TM) can 
10 display the Netshow (TM) compatible hypervideos created by the modified 
version of the authoring tool 1001 for Netshow (TM). In one embodiment, the 
modified version of the run-time module 1 1 01 for Netshow (TM) includes an 
ActiveX control, which facilitates embedding hypervideo in HTML pages and 
other applications supporting OLE/ActiveX controls. The ActiveX control in the 
1 5 modified version of the run-time module 1 1 01 for Netshow (TM) may make use 
of the Netshow (TM) ActiveX control to stream video. 

Modified Version of Authoring Tool for Netshow fTM^ 

The modified version of the authoring tool for Netshow (TM) may 

20 include the following functions. One or more .AVI and .MOV files may be 

imported into the modified version of the authoring tool for Netshow (TM) which 
may be used to create a hypervideo 10,007. In other embodiments, other types of 
media files, such as motion picture experts group (MPEG) files, may be used to 
create a hypervideo 1 0,007. Media objects can be defined and tracked, 

25 automatically or manually, in the visual media (e.g. the AVI and .MOV files). 
Media object properties, such as name, media object description, click and over 
cursors, and linked targets 7003 (e.g. URL, seek to frame, play another ASF file), 
may be defined for each media object. ASF markers (not to be confused with 
media object, or hotspot, markers) may be used to identify specified times or 

30 frames. 

Time-based events, including ASF script commands, (e.g. description, 
URL flip, play another ASF file) may be defined. Properties for an ASF file (e.g. 
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author, copyright, description, rating, title, bit rate, and window size) may be 
defined. Hypervideo data may be saved in a project file 1670. Video, and other 
multimedia data (e.g. audio and hypervideo data), may be exported into an ASF 
file. 

5 

GUI Design 

The modified version of authoring tool for Netshow (TM) 1 5,000 has a 
GUI 15,001, illustrated in Figure 15, that is similar to the GUI 10,001 for the 
authoring tool 1001 described above. In one embodiment, the GUI 15,001 of the 
1 0 modified version of the authoring tool for Netshow (TM) 1 5,000 may include the 
following GUI 15,001 components: menu bar 15,005, main tool bar 15,007, 
Media Warehouse window 15,003, Workshop Window 15,01 1 and two floating 
tool bars, the hotspot tool bar 1 5,008 and the workshop tool bar 1 5,009. 

The menu bar 15,005 may include the following headings and commands: 

15 File: 

New 

Open 

Save 

Save As ... 
20 Revert 
Send ... 

Import Media File 

Most Recently Used (MRU) List 

Exit 

25 

Edit: 

Undo 

Cut 

Copy 

30 Paste 
Delete 
Select All 
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Media: 

Large Icons 
Small Icons 
Report View 
5 Sort By... 

Export ASF 

Workshop: 
Play 

10 Pause 

Mark IN Frame 
Mark OUT Frame 
Select 

Define New Hotspot ... 
15 Auto Track 

Options: 

Settings . . . 

20 Window: 

Workshop Tool 
Hotspot Tool 
Close 
Close All 

25 

Help: 

Index 

Using Help 

About Authoring Tool (V-Active (TM)) 

30 

The main tool bar 15,007 includes buttons, which may be associated with 
menu bar 10.005 commands, and include: New, Open, Save, Revert, Undo, Cut, 
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Copy, Paste, Settings, About. The functions of the foregoing commands and 

buttons are known and understood by those skilled in the art. 

In one embodiment, the Media Warehouse window 15,003 may support 

media elements 1690 whose corresponding media files 1680 are either in AVI or 
5 QuickTime formats. The Import Media File dialog 2001 , thus, may only present 

*.aviand *.mov file selection. In other embodiments, the Import Media File 

dialog 2001 may display other types of media files, including MPEG video files. 
The Media Warehouse window 15,003 control bar will include new 

Export and Export All buttons. In one embodiment, the Relative Path Name 
10 button may be included in the Media window control bar. By actuating a switch, 

such as the right button of a mouse, while the pointer is over a thumbnail 

illustration 4007 in the Media Warehouse window 15,003, various commands, in 

a displayed menu, are displayed and can be executed. One command facilitates 

the display of a list of all the hotspots 8003 defined in the media corresponding to 
1 5 the thumbnail illustration 4007. This command may also permit the target(s) 

7003 of each hotspot 8003 to be illustrated. Other commands that can be 

actuated include: Export, Export as..., etc.. 

Upon placing the pointer on a thumbnail illustration 4007 in the Media 

Warehouse v^ndow 15,003, and doubling actuating (e.g. clicking), a switch (e.g. 
20 lefthand mouse switch), the Video Media Properties dialog will be displayed. 

The Video Media Properties dialog includes the General, Preview, Hotspots, 

Events, Markers and ASF properties sheets. 

In one embodiment, the Workshop window 8001 of the modified version 

of the authoring tool for Netshow (TM) 1 5,000 may include the features of the 
25 Workshop window 8001 described above, with the exception that: 

(1) the Preview button 2003 may be omitted; 

(2) the technique for linking target(s) 7003 to a hotspot 8003 may differ. 
In one embodiment, targets 8003 may only be defined in the Hotspot Properties 
sheet 7101. In this embodiment, targets may no longer be defined using a drag 

30 and drop technique with media elements 1 690 in the Media Warehouse window 
15,003; 
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(3) an indication of an ASF marker with its label may be displayed on top 
of the current working frame; and 

(4) ASF markers may be edited, removed and added by accessing the 
Media Warehouse window 15,003 only. 

5 In another embodiment, the Set Target button 901 5 may be present in the 

Workshop Tool bar to permit a target 7003 to be defined for a media object. 

Other user interfaces, including property sheets, of the GUI 15,001 of the 
modified version of the authoring tool for Netshow (TM) 15,000 will now be 
described. 

10 

Hotspot Properties Sheet 

The GUI 15,001 of the modified version of the authoring tool for 
Netshow (TM) 15,000 may include a Hotspot Properties Sheet 16,101, as 
illustrated in Figure 16. Each Hotspot Properties Sheet 16,101 is uniquely 
15 associated with a hotspot 8003. The Hotspot Properties Sheet 16,101 may 
include a Targets, or link, page 16,001 that facilitates linking target(s) 7003 to 
the hotspot 8003. 

The Targets page 16,001 may permit the following types of targets 7003 
to be linked to a hotspot 8003: 
20 a new ASF stream 1 6,201 to be played by the Netshow ActiveX (TM) 

control; 

flip URL 16,203; and 

seek to another position in video by fi-ame number 16,205 or ASF marker 

16,207. 

25 In one embodiment, a target 7003 may be browsed in a frame screen 16,901 on 
the Targets Page 1 6,1 01 . Specific frames or time base times in the target 7003 
may be displayed by specifying a corresponding ASF marker. The frame screen 
16,901 replaces the frame window 7901 used in the GUI 10,001 of the authoring 
tool 1001 described above. 

30 The General Properties page of the Hotspot Properties Sheet 16,101 may 

be similar to the General Properties page 7103 of the GUI 10,001 of the 
authoring tool 1001 described above. 
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Media Properties Sheet 

The GUI 15,001 of the modified version of the authoring tool for 
Netshow (TM) 15,000 may include a Media Properties Sheet 17,003, illustrated 
in Figure 1 7. The Media Properties Sheet 1 7,003 may be similar to the Media 
5 Properties Sheet 5003 of the GUI 10,001 of the authoring tool 1001 described 
above. 

In one embodiment, the Media Properties Sheet 17,003 may be modified 
to define time-based events, markers 1 155 and ASF file properties by including 
additional pages such as events, properties and markers pages. 

1 0 The Events page 17,101, illustrated in Figure 1 7, facilitates 

implementation of time-based events (e.g. ASF Script Commands). On the Events 
page 17,101, a frame, or time code, 17,103 and event 17,105 to be activated 
upon display of the frame 1 7,103 may be specified. The events that can be 
specified include change the ActiveX control's description 17,1 1 1 , URL flip 

15 17,1 13, and launch a new ASF file 17,1 15. This technique is similar to 
notification fi-ames 1 157 described above. 

The ASF Markers Page facilitates the assignment of ASF markers for 
specific fi-ames, or times in a time base. The ASF markers can also be viewed in 
the Workshop window 1 5,01 1 . 

20 The Media Properties Sheet 1 7,003 may include a Properties page 

18,001, illustrated in Figure 18. The Properties page 18,001 may be similar to 
the Details page 5001 of the GUI 10,001 of the authoring tool 1001 described 
above. The Properties page 18,001 facilitates definition of properties for ASF 
media files. The properties may include title 18,003, author 18,005, copyright 

25 1 8,007, description 1 8,009 and rating 1 8,01 1 . 
Exporting Media 

Hypervideo data and media may be exported by the authoring tool 1001 
into an exported file. For example, the modified version of the authoring tool for 
Netshow (TM) 15,000 may export the hypervideo data to an ASF file so that the 
30 ASF file includes both video and hypervideo data. In one embodiment, upon 
creating an exported file, a Properties page, for example for the ASF file 
properties, is displayed. The Properties dialog may be similar to the Properties 



wo 99/10822 



PCT/US98/17444 



95 

dialog used in the Microsoft Netshow Editor. The properties dialog may permit 
specification of bit rate properties and modification of ASF properties. 

In another embodiment, if the exported file is saved, a Save file dialog 
may appear. In a ftirther embodiment, an ASF file including video data and 
5 hypervideo data, including other authored data, e.g. markers, script commands 
and hotspots, is saved 

ActiveX (TM) Control of Modified Version of the Run-T ime Module for 
Netshow fTM^ 

10 The modified version of the run-time module for Netshow (TM) has an 

ActiveX (TM) control wrapped around the Microsoft (TM) Netshow (TM) On- 
Demand Player ActiveX (TM) control. As a result, hypervideos 10,007 in ASF 
can be displayed by the On-Demand Player ActiveX (TM) control. The ActiveX 
(TM) control of the modified version of the run-time module for Netshow (TM) 

1 5 may have the following functions: 

Open, play, stop and pause ASF files, including those ASF files created by 
the modified version of the authoring tool for Netshow (TM) 15,000; 

Signal the existence of a hotspot 8003 by changing the mouse pointer 
(e.g. over cursor) when the pointer is positioned over the hotspot 8003; 

20 Execute target(s) 7003 in response to the activation of a hotspot 8003. 

The target(s) 7003 may include: flip URL, seek to a position (e.g. frame) of a 
video, and play another ASF file; 

Activate time-based events (e.g. with ASF Script commands) including: 
change status bar description, flip URL, and play another ASF file; and Display 

25 properties of an ASF file. 

ActiveX rTM^ Control GUI for Modified Version of the Run^Time Module for 
Netshow rTM) 

In one embodiment, the ActiveX (TM) control for the modified version of 
30 the run-time module for Netshow (TM) may be an embedded control. The 
Active X (TM) Control for the modified version of the run-time module for 
Netshow (TM) GUI 1 9,001 , illustrated in Figure 1 9, may include one or more of 
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the following bars: a status bar 19,013, a display bar 19,0 11, and a control bar 
19,003. 

The status bar 19,013 may be used to display text, describing the status of 
the ASF streaming process. The status bar 19,013 may be changed using the 

5 change description event of ASF. The display bar 1 9,011 may indicate the 
current playing time of a video. The display bar 19,01 1 may be shown only in 
accordance with the ShowDisplay property. The control bar 19,003 may include 
a Play/Pause button 19,005, a Stop button 19,007 and a Time Slider 19,009. The 
ActiveX (TM) control for the modified version of the run-time module for 

1 0 Netshow (TM) GUII 1 9,00 1 may also have a properties dialog. 

When a pointer is over the Active X (TM) Control for the modified 
version of the run-time module for Netshow (TM) GUI 19,001, a pop-up floating 
menu may be displayed when a button, such as the right mouse button, is 
actuated, or clicked. The menu may include the following selections: Play, Pause, 

1 5 Stop; Display, Controls, Status bar; Properties; About. The Properties dialog 
box may resemble the Properties dialog box of the Microsoft (TM) Netshow 
(TM) On-Demand Player, and will allow setting the following properties: play 
count, auto rewind, auto start, display size, control display, buffering and 
streaming parameters. 



20 
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Functional Requirements of the ActiveX (TM) Control 

The ActiveX (TM) control for the modified version of the run-time 
module for Netshow (TM) 19,000 may have the following OLE/ActiveX 
properties, in addition to the typical properties of ActiveX controls: AutoRewind, 
5 Autostart, FileName, DisplaySize, PlayCount, ShowControls, ShowDisplay, 
ShowStatusBar, AnimationAtStart, InvokeURLs, TransparentAtStart, 
TransparentAtStop, StatusDescription. The ActiveX (TM) control for the 
modified version of the run-time module for Netshow (TM) 19,000 will consume 
relatively little memory, and will utilize the functionality of the Microsoft (TM) 
1 0 Netshow (TM) On-Demand Player ActiveX (TM) control around which the 
ActiveX (TM) control for the modified version of the run-time module for 
Netshow (TM) 19,000 is wrapped. 

2.41 Hvpervideo Authoring Tool for RealVideo (TM) 

15 A modified version of the authoring tool for RealVideo (TM) will now be 

described. RealVideo (TM) uses a proprietary format, not the ASF, utilizing 
Image Maps and Events files. The authoring tool 1001, described above, has 
been adapted to support the RealVideo (TM) Image Maps files and the 
RealVideo (TM) Events files, known to persons skilled in the art, to create 

20 interactive RealVideo (TM) files. The Image Map files and the Events files may 
respectively define hotspots 8003 and notification frames 1 157, as described 
above. 

In one embodiment, the modified version of the authoring tool for 
RealVideo (TM) may be designed to operate separately from the RealVideo 

25 (TM) run-time environment. In this embodiment, the modified version of the 
authoring tool for RealVideo (TM) need not be linked to the RealVideo (TM) 
run-time environment, other than by exporting hvpervideo data to the RealVideo 
(TM) Image Map or Events files whose data is used by the Real Video (TM) run- 
time environment. Thus, this embodiment may lack the capability to preview a 

30 hypervideo. 

In other embodiments of the modified version of the authoring tool for 
RealVideo (TM), other features of the authoring tool 1001 may be removed, or 
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modified to support the RealVideo (TM) run-time environment. In one 
embodiment, the modified version of the authoring tool for RealVideo (TM) has: 

No project preview; 

Limited media importing features; 
5 RealVideo (TM) Image Map export utility; 

RealVideo (TM) Events export utility; 

Limited target types 7003 corresponding to the RealVideo (TM) link, or 
target 7003, options; and 

No publish utilities for defining the path of the project file 1670 (e.g. .obv 

10 file). 



GUI Design 

One embodiment of the GUI 20,001 , illustrated in Figure 20, for die 
modified version of the authoring tool for RealVideo (TM) 20,000 may be similar 
1 5 to the GUI 1 5,001 of the modified version of the authoring tool for Netshow 
(TM) 15,000 described above. However, both GUIs 15,001, 20,001 may not 
necessarily be identical. 

In one embodiment, the GUI 20,001 of the modified version of the 
authoring tool for RealVideo (TM) 20,000 may include the following GUI 
20 20,001 components: menu bar 20,005, main tool bar 20,007, Media Warehouse 
window 20,003, Workshop Window 20,01 1 and two floating tool bars, the 
hotspot tool bar 20,008 and the workshop tool bar 20,009. 

The menu bar 20,005 may include the following headings and commands: 

File: 
25 New 
Open 
Save 

Save As ... 
Revert 

30 Send... 

Import Media File 
MRU List 
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Exit 



Edit: 

Undo 
5 Cut 
Copy 
Paste 
Delete 
Select All 

10 

Media: 

Large Icons 
Small Icons 
Report View 
15 Sort By... 

Export (RealVideo (TM) Image Map) file 
Export (RealVideo (TM) Image Map) file As... 
Export (RealVideo (TM) Image Map) file All 



20 Workshop: 

Mark IN Frame 
Mark OUT Frame 
Define New Hotspot . . . 
AutoTrack 

25 

Options: 

Settings... 



Window: 
30 Workshop Tool 

Hotspot Tool 
Close 
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Close All 

Help: 

Index 

5 Using Help 

About V-Active 

In one embodiment, the main tool bar 20,007 for the modified version of 
the authoring tool for RealVideo (TM) 20,005 may not include the Project 

1 0 Settings, Project View, Project Preview. 

In one embodiment, the modified version of the authoring tool for 
RealVideo (TM) 20,005 has a Media Warehouse Window 20,003. The Media 
Warehouse window includes a control bar 20,010. In another embodimenU only 
AVI & QuickTime media files may be supported by the modified version of the 

15 authoring tool for RealVideo (TM). In this embodiment, all other types of media 
files may not be imported into the Media Warehouse Window 1 0,003. Thus, the 
Import Media File dialog 2001 may only present *.avi and *.mov file selection. 
In this embodiment, other media file types may not be imported even if the "All 
Files option is used, or if an .obv file created by the authoring tool 1001 is 

20 used. In another embodiment, other media files, such as MPEG files, may be 
imported. 

In one embodiment, the Media window warehouse 10,003 conu-ol bar 
may include ^Export', and 'Export Air buttons. In another embodiment, the 
Relative Path Name button may be included in the Media window control bar. 

25 Upon actuating a switch, such as the right button on a mouse, while the pointer is 
over a thumbnail illustration 4007 of a media in the Media Warehouse Window 
10,003, a menu will be displayed. The menu options then can be selected, and 
include the commands Export and Export As... . Another menu option that can 
be selected displays a list of the hotspots 8003 in the media. 

30 In another embodiment, the modified version of the authoring tool for 

RealVideo (TM) 20,005 has a Media Workshop Window 20,01 1 . Many of the 
features of the media workshop window 8001 of the authoring tool 1001 may 
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also be found in the media workshop window 8001 of the modified version of the 
authoring tool for RealVideo (TM). For example, the menus 15,101, displayed 
when a switch, such as the right mouse button, is activated regardless as to 
whether the pointer is over or not over a hotspot, will remain the same. In one 
5 embodiment, the authoring tool and player for RealVideo (TM) may support the 
z-axis ordering of hotspots 8003, as described above. In one embodiment, the 
Preview button may not be used in the authoring tool for RealVideo (TM) 
20,005, for example in the workshop tool bar 20,101 . 

In yet another embodiment, the modified version of the authoring tool for 
1 0 RealVideo (TM) 20,005 has a workshop tool bar 20,009. Features of the Tools 
window 9001 of the authoring tool 1001 may also be found in the workshop tool 
bar 20,009 of the modified version of the authoring tool for RealVideo (TM) 
20,000. In one embodiment, however, the Set Target button 9015 may not be 
included in the workshop tool bar 20,009. In another embodiment of the 
1 5 workshop tool bar 20,009, the Ellipse Button 20,1 01 is used to draw circles 
rather than ellipses. 

In yet another embodiment, the GUI 20,001 of the modified version of the 
authoring tool for streaming video, including for RealVideo (TM), may exclude 
the following features found in the GUI 10,001 for the authoring tool 1001 
20 described above: 

Preview window 8101; 

Project View 1007; 

Project Settings Property sheet; 

Pages in the Media Properties Sheet 17,003 except the Properties page 

25 18,001; 

Preview sheet in the Settings dialog; 

Target Property page in the Hotspot Properties sheet; and 

All buttons and controls used to invoke the above features. 

30 Hotspot Properties Sheet 

In one embodiment, the GUI 20,001 of the modified version of the 
authoring tool for RealVideo (TM) may include a Hotspot Properties sheet 
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21,101, illustrated in Figure 21, similar, but not necessarily identical, to the 
Hotspot Properties sheet 16,101 described above. Each Hotspot Properties sheet 
21,101 is uniquely associated with a hotspot 8003, and may include a general and 
links, or targets, page. In one embodiment, however, the Hotspot Properties 
5 sheet 16,101 may not have a Target Properties page. In another embodiment, the 
over and click cursors may be dropped from the General page because the 
RealVideo (TM) system uses only the hand cursor. 

Targets 7003 required by the RealVideo (TM) run-time environment may 
be specified in the Target page 21 ,001 of the Hotspot Properties sheet 21,101 . In 

1 0 one embodiment, the Target page 2 1 ,001 may include at least the three types of 
targets 7003 that are executed upon actuation of a corresponding hotspot 8003, 
and are supported by RealVideo (TM). The targets 7003 include Seek 21,107, 
Player 2 1 , 1 03 and URL 21,105. 

The Seek 21,107 target identifies a frame number 21,108 in the video 

15 currently being displayed. Thus, when a corresponding hotspot 8003 is actuated, 
the video jumps to the frame having the frame number 21,108, and then that 
frame and successive frames are then displayed. In one embodiment, the video 
may be previewed on the Target page 21 ,001 . The frame corresponding to the 
frame number 21,108 may be selected during such previewing. When the frame 

20 is selected, a Mark button 21,111 may be actuated so that the corresponding 
frame number 21,108 is entered as the Seek 21,107 target. 

The Player 21,103 target is another multimedia stream whose location is 
defined by a stream URL 21,104. Thus, when a corresponding hotspot 8003 is 
actuated, the multimedia stream at the stream URL is executed, or performed. 

25 The URL option, a URL address must be identified. 

Media Properties Sheet 

The GUI 20,001 of the modified version of the authoring tool for 
RealVideo(TM) 20,000 may include a Media Properties Sheet similar to the 
30 Media Properties Sheet 1 7,003, illustrated in Figure 1 7. In one embodiment, the 
Media Properties sheet may, however, not include Defaults or Publish pages. In 
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another embodiment, an Events page, such as described above, may be added to 
the Media Properties Sheet. 

rnmpatibilitv 

5 In one embodiment, the formats of project files 1 670, associated with the 

authoring tool 1001 and the modified version of the authoring tools for 
RealVideo (TM) 20,001 and Netshow (TM) 15,001, may be compatible. Project 
files 1670 (e.g. .obv files) created by one authoring tool may be used by other 
authoring tools. Each authoring tool may use only the project file 1670 data 

1 0 corresponding to the features that the authoring tool supports. 

Rypnrtinp Image Map and Event Files 

Hypervideo data for RealVideo (TM) multimedia files may include both 
hotspot 8003 and notification frame 1 157 data. Upon authoring a hypervideo 

1 5 1 0,007, hotspot 8003 and notification frame 1157 data are initially stored in a 
project file 1670 (e.g. .obv file). In one embodiment, the project file 1670 will be 
associated with only one media, e.g. video, file. Subsequently, the modified 
version of the authoring tool for RealVideo(TM) 20,000 facilitates the 
exportation of the hotspot 8003 and notification ft-ame 1 157 data from the project 

20 file 1670 into separate files, respectively an Image Map file and an Event file. 
The Image Map file stores hotspot 8003 data. The Event file stores notification 
frame 1 157 data. The Image Map and Event files utilize RealVideo (TM) 
formats known to persons skilled in the art. 

In one embodiment, each RealVideo (TM) multimedia file may include 

25 only one video file. Thus, in this embodiment, the Image Map and Event files will 
only contain hotspot 8003 and notification frame 1 157 data for this sole video 
file. The modified version of the authoring tool for ReaiVideo(TM) 20,000 and 
associated files will now be further described in view of this embodiment. 

When exporting hotspot 8003 or notification frame 1 1 57 data to an Image 

30 Map or Event file, the path of the Image Map or Event file must be specified. In 
one embodiment, Export settings, including default path, can be defined using an 
Export page, or tab, in the OptionslSettings dialog. The Export page can also be 
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used to define the length of each frame sequence for each Image Map, further 
discussed below. The length of each frame sequence associated with an Image 
Map can be determined by corresponding transmission line bandwidth (e.g. 28800 
kbps, 33600 kbps, ISDN, etc.), and other parameters set in the authoring tool for 

5 RealVideo(TM) 20,000. 

Once a path has been specified, the Image Map or Event file can be 
created by invoking either the Export or Export As... commands. The Export 
command exports the ImageMap or Event file to the specified path. The Export 
As. . . command invokes an Export dialog in which the specified path may be 

10 redefined. An illustration of an exemplary Export dialog 22,001 invoked by the 
Export As... command is illustrated in Figure 22. The Export dialog 22,001 
invoked by the Export As ... command may include a browse button 22,003 and a 
thumbnail illustration 4007 of the video with which the Image Map or Event file 
is associated. The browse button 22,003, known to persons skilled in the art, 

1 5 permits perusal of other predefined paths for exported Image Map or Event files. 
In one embodiment, Map or Event files can be exported substantially 
simultaneously for all video files associated with a hypervideo 10,007 by invoking 
the Export All command. In one embodiment. Export dialogs 22,001 are 
displayed for the videos for which no Image Map path was previously defined. 

20 In one embodiment, the commands for exportation, described above, may 

be invoked in the following manners: 

Positioning the pointer over a thumbnail illustration 4007 of a specific 
media in the Media Warehouse window 20,003, and actuating a button, such as 
the Right mouse button to access the Export and Export As. . . commands; 

25 Actuating a button, such as an Export button 20,01 1 or an Export All 

button 20,012, in the control bar 20,010 in the Media Warehouse window 20,003 
to respectively invoke either the Export or Export All commands; and 

Invoking the Export, Export As and Export All commands in the 
media menu of the menu bar 20,005, 

30 

Transformation Process 
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Invocation of the Export commands causes the hotspot 8003 and 
notification frame 1 157 data in the project file 1670 to be transformed 
respectively into the formats of the Image Map and Event files. If a project file 
1670 (e.g. .obv file) is re-opened and edited after an initial transformation, or 
5 exportation, then transformation by invocation of the Export commands must be 
repeated. 

One embodiment of the transformation process for Image Map files will 
now be summarized. Unlike hotspot 8003 defined in a project file 1670, a 
hotspot 8003 in a Image Map file may only have a fixed geometry. Therefore, a 

1 0 hotspot 8003, whose geometry varies among frames, in a project file 1 670 must 
be redefined as two or more hotspots 8003, each of whose geometry is fixed, in 
the corresponding Image Map file. A hotspot 8003, defined in the Image Map 
file, may have a fixed geometry over two or more frames. 

Hotspot(s) 8003 will not be translated by the invocation of an Export 

1 5 command if a hotspot Enable flag is set off in a General page 2 1 ,002 of a 

corresponding Hotspot Properties sheet 21,101. Thus, such hotspot(s) 8003 will 
not be included in the Image Map file created by the transformation process. The 
default length of those frame sequences will be determined in accordance to the 
transmission bandwidth, as described above. A new MAP range of frames will be 

20 used for the first and last frames in which a hotspot 8003 appears to ensure that 
the hotspot 8003 is only provide for frames in which the hotspot 8003 was 
originally defined. The number of MAP statements created by the modified 
authoring tool for RealVideo (TM) 20,001 is determined by the available 
bandwidth of the client-server network used to perform the hypervideo 10,007. 

25 

Creation of RealVideo (TM) Multimedia Files includi np Hvpervideo Data 

In one embodiment, each Image Map or Events file must then be encoded 
into an .rm file using the RealVideo (TM) rmmerge tool. The resulting .rm file 
must then be encoded, or exported, into a corresponding, already encoded, media 
30 (e.g. video) file again using the rmmerge tool to create a final .rm file. In another 
embodiment, the foregoing process is performed by the user who must twice run 
the rmmerge tool. 
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In a further embodiment, this process is automated. The modified version 
of the authoring tool for RealVideo (TM) 20,000 not only exports the hotspot 
.-8003 and notification frame 1 157 data, as described above, but also causes the 
rmmerge tool to be executed so as to create the final .rm file. To execute the 
5 rmmerge tool, the modified version of the authoring tool for RealVideo (TM) 
20,000 utilizes dynamic Hnk libraries (DLLs), rmimap.dll and rmevents.dll, and 
the RealVideo (TM) software development kit (SDK). The modified version of 
the authoring tool for RealVideo (TM) 20,000 utilizes the RealVideo (TM) SDK 
to encode the video file into a RealVideo (TM) .rm video file format. 
1 0 When implementing the automated process, an export dialog for 

automatic translation 23,001, illustrated in Figure 23, may be used. The export 
dialog for automatic translation 23,001. is displayed upon executing an export 
command. 

The export dialog for automatic translation 23,001 permits either 

1 5 temporary or permanent .rm files including Image Map and/or Event file data to 
be created. Permanent .rm files for Image Map and Event file data may be created 
upon checking the Use intermediate files box 23,1 01 . Upon actuating the create 
file/s button 23,1 03, permanent .rm files may be created for Image Map and 
Event file data may be files if their respective Build boxes 23,105, 23,107 are 

20 checked. To create the permanent .rm files including Image Map and Event data, 
temporary Image Map and the Events files are first built as temporary text files. 
Then, the permanent .rm files are created for the Image Map and Events file data, 
using the rmmerge tool. Respective names 23,109, 23,1 1 1 for the permanent .rm 
files for Image Map and Events data are specified in the export dialog for 

25 automatic translation 23,001 . An encoded video .rm file is also created from the 
original video file using the RealVideo (TM) SDK only if the build box is 
checked. Because a video file may take minutes to encode, the build box may be 
left unchecked to avoid unnecessarily repeating video file encoding every time 
hotspots 8003 are encoded. 

30 The name 23,1 1 7 for the encoded video .rm file is specified in the export 

dialog for automatic translation 23,001 . After the permanent .rm file(s) are built, 
the video .rm file is merged with the permanent .rm file(s), using the rmmerge 
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tool, if at least one corresponding Merge box is checked, to create the final .rni 
file. The name 23,1 19 for the final .rm file is specified in the export dialog for 
automatic translation 23,001 . If neither Merge check boxes 23,1 13, 23,1 1 5 is 
checked, then the final .rm file is not created. 
5 Alternatively, if the Use intermediate files box 23,101 is not checked, the 

video file is encoded into a temporary .rm file using the RealVideo (TM) SDK. If 
the Image Map file Merge box 23,1 13 is checked, then the Image Map data is 
translated into a temporary text file, which is translated into a corresponding 
temporary .rm file. If the Events Merge box 23,1 1 5 is checked, then the Events 
1 0 data is translated into a temporary text file, which is translated into a 

corresponding temporary .rm file. Then, the video .rm file is merged, using the 
rmmerge tool, with the specified temporary ,rm files to create the specified final 
.rm file. If both of the Merge check boxes are unchecked, the video file is simply 
encoded into the final .rm file. 

15 

Fncndinp Properties 

In one embodiment, the export dialog for automatic translation 23,001 

may include an Encoding Properties Sheet 24,000, illustrated in Figure 24A. The 

Encoding Properties Sheet 24,000 permits entry of the data required by the 
20 RealVideo (TM) Encoder. The Encoding Properties Sheet 24,000 may be 

accessed by actuating the Encoding Properties... button 23,121 . 

The Encoding Properties Sheet 24,000 includes the Encoding Data page 

24,001, illustrated in Figure 24A, and the Encoding parameters page 24,002, 

illustrated in Figure 24B. In another embodiment, the Encoding Data page 
25 24,001 and the Encoding parameters page 24,002 may be added to the Media 

Properties Sheet. 

The Encoding Data page 24,001 may contain general data that will be 
inserted into the video .rm file. The general data may include the title 24,003. 
authorship 24,005, and copyright information 24,007 associated with the 
30 hypervideo 10,007 created in the RealVideo (TM) format. 

The Encoding Parameters page may contain parameters for the video 
encoding process, including: 
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a Template box 24,101 from which standard RealVideo templates can be 
selected. By selecting a standard template, other data fields on the Encoding 
Parameters page 24,002 may be selected according to the data associated with 
the selected template. 
5 Changing one of the other fields will set the template field to the 

appropriate value (if there is no template with current parameters. Custom will be 
set). 

Multiple Video Files 

1 0 The modified version of the authoring tool for RealVideo (TM) 20,000 

may be used to define and manipulate hotspots 8003 and notification frames 
1 1 57, including defining target(s) 7003, in multiple video media files substantially 
simultaneously. The hoispot 8003 and notification frame 11 57 data may then be 
exported into final .rm files corresponding to each of these video files, using the 

1 5 process described above, in a manner similar to the Save/Save All/Save As. . . 
feature in a multi-document application. 

2.4 Dynamic Hypervideo 

20 Previously, hypervideos 10,007 have been suggested as having fixed or 

static parameters. Static hypervideo 1 0,007 parameters can not be changed 
during the performance of the hypervideo 1 0,007. One example of a static 
project file parameter is the parameter that defines a target 7003 media file linked 
to a media object. Thus, the target 7003 media file can not be changed during the 

25 performance of the hypervideo 10,007. As a result, a viewer of a performing 
hypervideo 10,007 is limited to navigating through media files defined during 
hypervideo 10,007 authoring. 

To enhance the robustness and versatility of hypervideos 10,007, it is 
desirable to create dynamic hypervideos by allowing the hypervideo 1 0,007 

30 parameters to vary during hypervideo 10,007 performance. Thus, for example, 
the hypervideo story board 1116 may change during hypervideo performance. 
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Dynamic hypervideos are hypervideos 10,007 having hypervideo 10,007 
parameters thai may vary during hypervideo 10,007 performance. The 
hypervideo parameters may include, but are not limited to, parameters describing 
media objects (e.g. hotspots 8003 and notification frames 1 157), targets 7003, 
5 target levels 7601 , cursors, and hypervideo 1 0,007 display window size. For 
example, by varying parameters, targets 7003 of media objects may be changed, 
media objects may be disabled so that the media objects may not be actuated, and 
hypervideo 10,007 display window size may be altered during hypervideo 10,007 
performance. 

1 0 The hypervideo 1 0,007 parameters may be varied during hypervideo 

1 0,007 performance as a result of communications from application programs, 
and data retrieved from databases or log files. Viewer interactions may also 
indirectly vary hypervideo 10,007 parameters as described below. Viewer 
interactions with the hypervideo 10,007 may occur, for example, when a viewer 

1 5 actuates a hotspot 8003, or just positions a pointer over a hotspot 8003. 

Dynamic hypervideos that communicate with application program(s) may be 
knovsoi as hypervideo applications. Hypervideo applications may be used for 
example for electronic commerce and corporate training, and will be further 
described below. 

20 In one embodiment, dynamic hypervideos may be created with a 

hypervideo system 25,000, depicted in Figure 25. The hypervideo system 
25,000, illustrated in Figure 25, includes multiple servers, each having one or 
more components (e.g. database 25,037, project file 1670, application program 
25,038, etc.) A client 25,003, having a player 25,039, may be coupled by a first 

25 network 25,050a to a first server 25,005a that may have a dynamic hypervideo 
server 25,035, logging tools program 25,047, log file 25,049 (e.g. a log database 
and/or a log file) and a project file 1 670. The client 25,003 is also coupled by a 
second network 25,050b to a second server 25,005b that may have a media 
server 25,045. The media server 25,045 may include a video server and/or a web 

30 (or HTML) server. The first server 25,005a is also coupled by a third network 
25,050c to a third server 25,005c that may have application program(s) (e.g. third 
party applications and/or third party servers) 25,038. The first server 25,005a is 
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also coupled by a fourth network 25,050n to a fourth server 25,005n that may 
have database(s) (e.g. database severs) 25,037. In one embodiment, the player 
includes a modified run-time module for receiving and processing streamed 
hypervideo data, and a media player (e.g. RealNetworks (TM) media player) for 

5 receiving and projecting streaming audio and video to a viewer. In one 
embodiment, the modified run-time module controls the media player. 

In another embodiment, each network 25,050a-25,050n may use the 
Internet or an intranet, or a combination thereof In yet another embodiment, the 
hypervideo system 25,000 may include multiple clients 25,003 coupled to the first 

10 and second servers 25,005a, 25,005b. The additional clients may each include a 
player 25,039. In one embodiment, an additional client 25,003 may be coupled to 
the first server 25,005a, and includes a modified authoring tool for creating a 
dynamic hypervideo. In one embodiment, the modified authoring tool may be a 
modified version of the authoring tool 1001, described above, or a variant 

15 thereof. The modified authoring tool may be used to create the project file 1670 
that can be transferred to the first server 25,005a electronically across a network 
or physically with a storage device, such as a compact disk-read only memory 
(CD-ROM). 

In further embodiments, the hypervideo system 25,000 may have fewer or 
20 more servers upon which the components are respectively consolidated or further 
distributed. In another embodiment the servers and client(s) may be 
interconnected in a different manner, for example directly coupled to one another. 
In yet a fiirther embodiment, the database(s) 25,037 may include multiple 
databases, each of which operates with one or more components of the 
25 hypervideo system 25,000. 

One embodiment of the operation of the hypervideo system 25,000 will 
now be described. In this embodiment, the video and hypervideo data streams 
may be wholly separated. For example, hyper\'ideo data may be streamed from 
the dynamic hypervideo server 25,035 to the client 25,003. Video data may be 
30 separately streamed from the media server 25,045 to the player 25,039. 

In another embodiment, some hypervideo data may be exported into a 
multimedia (e.g. video) file by a modified authoring tool. The media and 
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hypervideo data in the multimedia file may be streamed from the media server 
25,045 to the player 25,039. For example, the media server 25,045 may stream 
hypervideo data, including media object identifier data, time or fi-ame data, shape 
data, and geometry data (e.g. layer one of multilayer hypervideo data stream 
5 described above), to the player 25,039. This technique may increase second 
network 25,005b transmission capacity consumption, but may reduce first 
network 25,005a transmission capacity consumption and dynamic hypervideo 
server 25,035 processing requirements. 

In one embodiment, some hypervideo data may be processed in the 

10 dynamic hypervideo server 25,035, and not streamed to the client 25,003, to 
reduce the player's processing and memory requirements, and to efficiently 
facilitate dynamic hypervideos. The interaction between the player 25,039 and 
the dynamic hypervideo server 25,035 will now be further illustrated. 

In one embodiment, the client 25,003 communicates with the dynamic 

15 hypervideo server 25,035 by firing events, further described below, to a dynamic 
hypervideo server 25,035. Upon receiving the fired event, the dynamic 
hypervideo server 25,035 may respond by sending a command, which may 
include hypervideo data, to the client 25,003. For example, if a viewer actuates a 
hotspot 8003 during the performance of a media in the player 25,039, the client 

20 25,003 may fire an event to the server indicating that the hotspot 8003 has been 
actuated. In response, the dynamic hypervideo server 25,035 may stream a 
command, including target 7003 data identifying video files to be executed, to the 
player 25,039. The player 25,039 may then stream a request to the media server 
25,045 that the identified video files be streamed to the player 25,039 so that the 

25 video files may be performed. Other techniques for varying project file 
parameters will be later described. 

Dvnamic Functions Provided bv the Hvpervideo System 

Dynamic functionality provided by a hypervideo system 25,000 will now 
30 be fiirther described. Hypervideo systems 25,000 may provide functions found in 
hypervideos 10,007, described above. Thus, like hypervideos 10,007, dynamic 
hypervideos may include hotspots 8003, event notification 1114, and markers 
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11 55. Additionally, however, these features may be modified during the 
performance of the dynamic hypervideo, for example, based upon information 
derived from the database 25,037, application program 25,038, a log file 25,049, 
or user interaction. 

5 Dynamic functions provided by the hypervideo system 25,000 may 

include: 

a. Hotspot Feature Availability. Hotspot 8003 features may be 
enabled or disabled according to data retrieved from a database 
25,037, an application program 25,0389, a log file 25,049, or due 

10 to viewer interaction. Different hotspots 8003 features may be 

affected, for example, visual marking of hotspots 8003 or 
execution of target(s) 7003 may be disabled. 

b. Targets. Targets 7003 linked to hotspots 8003 may be modified 
by removing or adding tsirgets 7003, or by changing the execution 

1 5 and display parameters of target(s) 7003. For example, a dynamic 

hypervideo may be an interactive advertisement for a department 
store which may be viewed by customers. The dynamic 
hypervideo server 25,035 may ascertain the identity of customers 
viewing the dynamic hypervideo. The dynamic hypervideo server 

20 25,035 may also query a database 25,037 to determine whether 

the customer holds the department store's credit card. The 
customers may point and click on an illustration of department 
store's credit card. Based upon the query results, the dynamic 
hypervideo server 25,037 may cause monthly special offers to be 

25 displayed to customers that are credh card holders. Also based 

upon the query results, the dynamic hypervideo server 25,037 may 
cause information illustrating the privileges of a holder of the 
department store credit card, and about how the credit card may 
be acquired, to customers who do not hold the department store's 

30 credit card. 
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Project Settings. General attributes of the dynamic hypervideo, 
including initial settings, such as base target(s) 1 1,001 and display 
parameters, may be modified. 

Media Properties. Media elements 1690 of the dynamic 
hypervideo may have general properties unrelated to targets. 
Such properties may include: 

1) Audio track. Videos may include corresponding audio 
spoken in different languages and thus stored separately, 
for example, on different tracks. For example a dynamic 
hypervideo performing in a French speaking environment 
may adapt to broadcast French language audio; and 

2) Restricted access videos and video versions. Based upon 
externally supplied information, a dynamic hypervideos 
may ascertain which, if any, version of a media, such as a 
video, a viewer is authorized to observe. For example, a 
dynamic hypervideo configured for children may only 
display appropriately rated versions of media. 

Markers. Markers 1 155, which may be used to highlight media 
objects, or may convey additional information associated with the 
media object, may be modified. Such markers may be overiay 
images, text, or special filters. For example, a training hypervideo 
application may include a hypervideo simulation of an industrial 
plant. The dynamic hypervideo may display relevant information, 
such as temperature and humidity, as markers 1 155 next to 
equipment in the industrial plant. The markers 1 155 may display 
the temperature and humidity data in real-time. This is desirable 
because, for example, in the industrial plant simulation, the 
temperature may constantly change in response to trainee 
interactions with the dynamic hypervideo. The marker 1 155 
displaying temperature data may obtain the temperature 
data from a database 25,037. 
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Dynamic hypervideos may also support notification frames 1 157. 
Notification frames 11 57, or time based events, may be implemented in media, 
including hypervideo applications. Notification frames 1 157, as described above, 
permits information, including text, graphics and video, to be displayed in 
5 synchronization with a timebase (e.g. by frame unit or time unit) of a video to 
enrich the information content of a dynamic hypervideo. The hypervideo system 
25,000 may execute a target 7003 when a hypervideo 1 0,007 reaches a certain 
time or frame (e.g. notification frame 1 157). A target 7003 may include any of 
those previously or subsequently described, including an image, a video, or an 
1 0 executable program. Thus, for example, while a hypervideo application is 

displayed, slides containing relevant information may be displayed concurrently 
with notification frames 1 1 57. In one embodiment, notification frame 1 1 57 target 
or frame parameters may be altered during the performance of a hypervideo. 

15 Dynamic Data Sources 

Data for dynamic project file parameters may be provided from the 
following sources: 

Databases . Commonly available databases may be implemented in the 
20 hypervideo system 25,000. However, the hypervideo system 25,000 may 

be modified to accommodate other databases. Object Database 
Connectivity (ODBC), a standard windows applications program 
interface (API), may be used to facilitate communications, for example, 
between the dynamic hypervideo server 25,035 and a database 25,037. 
25 The hypervideo database may obtain data from a database 25,037 by 

issuing a query to the database 25,037. During query authoring, 
placeholders (e.g. variables) whose contents are determined during 
hypervideo application execution, may be used in the queries. In one 
embodiment, the queries may be standard query language (SQL) queries. 

30 

A pplications . Hypervideo applications may be implemented with the 
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hypervideo system 25,000 including an applications program 25,038. In 
one embodiment, the application program(s) 25,038 may receive and/or 
generate data in real-time. Other hypervideo system 25,000 components, 
including the dynamic hypervideo server 25,035, may serve as sources 
and/or destinations for such data associated with the application 
programs(s) 25,038. In one embodiment, application programs(s) 25,038 
may directly communicate with the hypervideo system 25,000 
components through SDK-like interfaces. Such communications may be 
facilitated with ActiveX commands and events, and scripting languages 
such as VBScript or JavaScript. Communications between application 
program(s) 25,038 and other hypervideo system 25,000 components may 
be stored and retrieved from an intermediary file. In one embodiment, 
data may be transferred between application programs(s) 25,038 through 
a database 25,037, described above. 

In one embodiment, an application program 25,038 may send commands 
or data to a dynamic hypervideo server (and thus to a hypervideo 1 0,007) 
through call back objects or APIs. The application program 25,038 
commands or data may be used to manipulate project file parameters 
during dynamic hypervideo performance. The dynamic hypervideo may 
communicate with the application program 25,038 by firing events, for 
example through a call back object, to the application program 25,038. In 
another embodiment, a player 25,039 and an applications program 25,038 
may communicate with one another directly, if a network connection 
exists, or indirectly through the dynamic hypervideo server 25,038. 

A hypervideo application may also include one or more application 
programs 25,038 that generate graphical output prepared for display. The 
hypervideo application may access the application program(s) 25,038 to 
obtain the graphical output, and may do so, in a manner similarto the one 
described above, using placeholders. 
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If the graphical output is available, it may then be displayed. An example 
of this technique will now be described. A kiosk may include a 
hypervideo system 25,000 for performing a dynamic hypervideo 
application that provides worldwide weather forecasting information. The 
5 dynamic hypervideo application may display a turning globe. A viewer 

may point and click on a region of the globe. Then, the dynamic 
hypervideo application may query a weather forecasting database to 
obtain next week's forecasted weather data for the selected region. The 
retrieved forecasted weather data may then be formulated in a bar chart by 
10 a charting application. The dynamic hypervideo application then may 

automatically display the bar chart over the selected region of the turning 
globe. 

Log File The hypervideo system 25,000 may permit hypervideo 10,007 
15 activities to be logged, by the dynamic hypervideo server 25,035, during 

hypervideo 10,007 performance. In one embodiment, an author of a 
hypervideo application may specify, during hypervideo application 
authoring, which viewer interactions (or other hypervideo application 
activities) and corresponding information are to be logged. During 
20 hypervideo application performance, the logged information may be 

retrieved by the dynamic hypervideo server 25,035 to manipulate the 
performance of the hypervideo application. 

Hypervideo Applications 
25 Private, governmental and commercial organizations are becoming reliant 

upon network application programs that facilitate interaction within or outside 

the organizations. In order to make the application programs more compelling 

and intuitive to users, the application programs should include dynamic content. 

For example, application programs may be enriched by including visual media, 
30 including graphical and animation. Application programs may be made more 

intuitive to users by allowing the users to interact with the visual media, such as 

with videos. 
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With hypervideo applications viewers can navigate and access a variety of 
databases, forms, etc., including knowledge databases, purchasing forms, 
instructional material or any other services provided by the application. 
Exemplary hypervideo applications include: 

5 

Electronic Commerce Application 

Businesses, ranging from small retailers to large manufacturers, are 
availing themselves of the Internet and other networks to promote, market and 
sell their services and products. Electronic commerce solution providers offer 

1 0 toolkits for developing and managing on-line web sites, stores and catalogs. Such 
toolkits may facilitate storefront creation, transaction processing, and electronic 
distribution of software or data. Lack of potential customer interest in sites 
created with such toolkits has lead businesses to look for more compelling web 
sites. Video can be used to make such web sites more alluring to enhance 

1 5 potential customer interest and enhance product promotion. 

However, it may be desirable to closely integrate the video with the 
purchasing process. For example, it may be preferable for a potential customer to 
point and click on an item (e.g. media object) in a hypervideo 10,007 to initiate 
the purchase of the displayed item. 

20 The purchase process may be implemented with a hypervideo application 

provided by the hypervideo system 25,000 including an electronic commerce 
application program. In one embodiment, the hypervideo application may be 
dynamic. A web site may include the dynamic hypervideo application whose 
parameters may vary, for example depending upon date or time, or depending 

25 upon the customer accessing the web site. Further, the hypervideo system 25,00 
may record the activities performed by customers during their visits to the web 
site. 

An exemplary electronic commerce hypervideo application will now be 
described. Potential customers may view and interact with a hypervideo 
30 application on a web site of a clothing store. The hypervideo application displays 
a fashion show in which models exhibit garments while walking down a runway. 
A potential customer may point and click on a garment worn by a model to obtain 
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information associated with the garment, and to obtain an order form which can 
be completed to buy the garment. 

The hypervideo application may be dynamic. The information, displayed 
on a player 25,039 when a potential customer points and clicks on a hotspot 

5 associated with a garment, may vary in real-time. For example, the price of the 
garment may be varied in real-time. When the potential customer points and 
clicks on the garment hotspot, the dynamic hypervideo server 25,035 may query a 
database 25,037 including the garment prices. The garment prices in the database 
25,037 may be varied in real time. If a potential customer positions a pointer 

1 0 over the garment hotspot at least a certain number of times (causing event 
messages to be fired from the player 25,039 to the dynamic hypervideo server 
25,035 each time the pointer is positioned over the garment hotspot), the 
dynamic hypervideo server 25,035 (upon receiving at least the certain number of 
event messages) may cause a discounted sales price to be displayed for the 

1 5 garment. The dynamic hypervideo server 25,035 may then facilitate the 
customer's purchase of the garment at the discounted sales price . 

Training A pplication 

Many organizations utilize Intranets to provide training services to their 
20 employees, or other parties. Parties may participate in prepared lessons, practice 
drills, or view live demonstrations. Video is already being used in training, for 
example to display live demonstrations. 

Hypervideo applications, however, enhance the allure and accuracy, and 
hence the effectiveness, of training. An exemplary hypervideo application for 
25 training will now be described. A trainee may view a hypervideo application that 
displays an experiment conducted in a laboratory of a chemical plant. The trainee 
may point and click on one of the chemicals reacting during the experiment. As a 
result of the trainee's action, data obtained by the dynamic hypervideo server 
25,035 from a database 25,037 or an application program 25,038 at that point in 
30 time during the experiment may be displayed. 



Media-Inten sive Application 
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Organizations, for example entertainment businesses, news providers, and 
government agencies, may store large quantities of video which may be retrieved 
over networks, such as intranets, the Internet, or combinations thereof. In one 
embodiment, the videos may be stored in a video database 25,05 1 . For example, 
5 news services may store numerous videos of past and present news stories on 
servers. A hypervideo application may be formed from the videos to permit 
viewers to interact with the 'videos'. For example, a viewer may point and click 
on a person or item displayed in a video so that fiirther information, such as a 
snapshot or related articles, about the selected person or item is displayed to the 
10 viewer. 

Interactive Advertpsppg Applicatioin 

Interactive advertising, for example online advertising, has unique 
capabilities in comparison to other forms of advertising. Viewers of interactive 

15 advertising may readily access further information about an advertised product or 
service, or conversely readily ignore such advertising. Further, interactive 
advertising applications may record data relevant to advertising effectiveness, 
including the type and frequency of information sought by viewers. For example, 
the recorded data can be used to bill advertisers based upon the information 

20 displayed to the viewers. 

Interactive advertising may be implemented in a more effective and 
alluring manner with dynamic hypervideo applications. An online advertisement 
hypervideo application may permit interactive advertisements to be altered in 
response to viewer actions. 

25 An exemplary dynamic hypervideo advertising application will now be 

described. While watching an advertisement, such as an automobile 
advertisement, a viewer may point and click on a hotspot associated with a car, 
shown driving along a picturesque shoreline highway, to obtain further 
information about the car. Other targets, executed when the car hotspot is 

30 actuated, may include a purchase order form or a chart of current revenue figures 
related to sales of the car model. 
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Multi-Purpose Application 

Hypervideo applications may have multi-purposes, for example electronic 
commerce, advertising, and training. An illustration of a multi-purpose dynamic 
hypervideo application follows. 
5 A kitchen appliance retailer implements an online store to sell kitchen 

appliances. Their web site permits viewers to look at and purchase items used in 
a kitchen, including cutlery and cabinet sets. The web site includes: 

1 . Interactive sections, or attractions, including a recipe comer in 
which renowned chefs demonstrate cooking their favorite dishes, and the utensil 

1 0 comer in which cooking utensils are demonstrated. 

2. Knowledge database of kitchen appliances and other kitchen 
related issues. 

3. Purchasing facilities permitting on-line purchasing of culinary 
goods. Payment and billing may be performed with a credit card on-line, or 

1 5 through the mail. Purchased goods may be mailed to the customers. 

4. A home page, for regular customers of the kitchen appliance 
retailer, where the regular customers may obtain special purchase offers, or 
receive guidance about designing their kitchens. 

5. Kitchen-zine. An interactive e-zine about cooking and kitchens. 
20 linked directly to the web site's purchasing facilities. 

Another service for regular customers may also include creation and maintenance 
of online wedding gift lists. 

The kitchen appliance retailer's computer system may include aa database 
25,037, a system administrator, a media server 25,045, an internal local area 

25 network (LAN) 25,050, and a dynamic hypervideo server 25,035. 

The interactive recipe comer may include streaming hypervideos of chefs 
preparing their favorites dishes. The hypervideos 10,007 may be embedded inside 
HTML pages, which may include more information about the corresponding 
recipe and the final product. The HTML page may change as dish preparation 

30 proceeds. Pointing and clicking on a product found in the hypervideo 10,007 
may cause more inforaiation to appear (e.g. pop-up) about the product. Pointing 
and clicking on an appliance may cause a query of the retailer's database to obtain 
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information about the appliance. The information may be displayed to the user as 
an HTML page in a frame separate from the video. The appliances, found in the 
retailer's current inventory, may be marked (e.g. highlighted) in the hypervideo 
application. Appliances which are on special sale may be marked with SALE 
5 tags. Pointing and clicking on a hotspot corresponding to an appliance may 
display, instead of the originally viewed media, a demonstration video of the 
appliance. The actuation of the appliance hotspot may also cause another target 
7003 to be executed. For example, a URL flip may be executed to display details 
about the selected appliance in another frame. The newly displayed video may 
10 also include a small image on it reciting BUY NOW. Upon pointing and clicking 
upon the BUY NOW image, an order form for the corresponding appliance may 
be displayed to permit the viewer to purchase the appliance from the retailer. 

Other Hvpervideo Application Uses 

15 Hypervideos 10,007 and hypervideo applications may be used to provide 

interactivity for the following: corporate training, distance/web-based learning, 
multimedia display kiosks and point of sale terminals, video archive retrieval, 
electronic commerce (e.g. online stores), security systems, medical analysis and 
treatment systems, entertainment systems, advertising, hotel information 

20 provision, real estate sales, architectural design demonstration, industrial 

tele-operation, home video display, adult entertainment display, customer support 
automation, catalogs, sales force automation, and corporate communications. 

Database Integration and External Data Referencing 

25 In one embodiment, data for dynamic hypervideo parameters may be 

extracted from a database 25,037. Data extracted from database(s) 25,037 and 
application(s) 25,038 may be in a raw format, such as binary numbers or text, or 
in a processed format, such as graphics or a video. In one embodiment, data 
from external sources (e.g. database(s) 25,037, application program(s) 25,038, 

30 log file(s) 25,049), may be integrated into a hypervideo application in a two step 
process. First, data from an external source may be accessed, and relevant data 
retrieved by the hypervideo system 25,000. Second, the retrieved data can either 
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be processed for display, or provided, for example to the dynamic hypervideo 
server 25,035, to control dynamic hypervideo performance. 

In another embodiment, a hypervideo 10,007 may be used as an interface 
to permit a viewer to access data from a database in an intuitive manner. For 
5 example, in a hypervideo application of a clothing store, dresses, for which 
hotspots 8003 are defined, may be marked with tags displaying their price. Such 
prices may updated in real time through the clothing store's database. In another 
example, a web site may provide support for truck engine mechanics. Videos of 
truck engines may be displayed through the web site. The videos include 

10 hotspots 8003, which allow viewers, such as truck engine mechanics, to point and 
click on truck engine parts to obtain information about those parts. The site 
maintains a database of registered truck engine mechanics. Only truck engine 
mechanics that match the web site's registrations may be allowed to obtain 
information about the parts, linked to hotspots associated with the parts, and 

1 5 displayed when the registered truck engine mechanics point and cHck on 

corresponding hotspots 8003. Others, not identified in the database, may only 
view the video, and may not access the part information. 

Database Access During Authoring 
20 In one embodiment, an authoring tool 1001 may access the database 

25,037, including table structures and data, when a hypervideo application is 
authored. In another embodiment, the authoring tool 1001 may partially access 
the database 25,037 so that table structures, but not data, are available to the 
authoring tool 1001 when a hypervideo application is authored. 

25 

Integration of A Dynamic Hypervideo with Multimedia and Video Databases 

Databases including multimedia and video are special types of external 
databases, and may also be used in a hypervideo 1 0,007. Databases, including 
multimedia and video databases, may be accessed to dynamically insert new 
30 media in a hypervideo 1 0,007 during the performance of a hypervideo 1 0,007. 
Videos or pictures, that were not originally authored in a hypervideo 10,007, may 
be displayed during the performance of the hypervideo 10,007. 
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For example, a broadcasting network having a large database of reportage 
video, in which images of persons are marked. A viewer may point and click on a 
person's image to access the broadcasting network's database to retrieve the 
most recently available photograph of the person. The photograph may then be 
5 displayed side by side with the currently playing hypervideo 1 0,007. Multimedia 
and video database data types, used in hypervideo 10,007s, may include video, 
images, sound, music, cursors, icons, etc. 

Database Storage of Hvpervideo Data 

10 In one embodiment, hypervideo 10,007 data may be stored in a database, 

such as a conventional database, rather than in a project file 1670 which may use 
a proprietary format. As a result, the hypervideo 10,007 data, such as the 
hypervideo 1 0,007 data corresponding to media and media objects, may be 
readily accessed, for example by a standard query language (SQL) query. The 

1 5 hypervideo 1 0,007 data in a database may be accessed and manipulated to modify 
hypervideo 10,007 performance and/or to permit logging or the acquisition 
statistics pertaining to hypervideo 10,007 performance. The hypervideo 10,007 
statistics may be derived from user interactions with a hypervideo 10,007. The 
hypervideo 10,007 data stored in a database 25,037 may include the range of 

20 frames in which a hotspot 8003 is defined, location of notification frames 1 157 in 
a video, the number of video frames displayed, or the number of times a hotspot 
8003 has been actuated. 

Data Lpgging and Analysis 
25 In another embodiment, the hypervideo system 25,000 may also log 

information about and during the performance of a hypervideo 10,007. To this 
end, the hypervideo system 25,000 includes logging tools 25,047 that permit: 

1 . Logging requirements for a hypervideo 1 0,007 to be defined. 

2. Logging information to be analyzed during or after the performance of the 
30 hypervideo 10,007. When analyzed during the performance of the hypervideo 

10,007, the logging information may be used to alter the parameters of the 
hypervideo 10,007. 
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3. Analyzing offline statistics about logging information gathered during 
hypervideo 1 0,007 performance (e.g after or during hypervideo 1 0,007 
performance). 

In one embodiment, when a hypervideo 10,007 is authored, the logging 
5 requirements, or data to be logged, may be specified. The data to be logged may 
be specified by the authoring tool 1001, or with logging tools 25,047. The 
logged data may include the hypervideo 10,007 activities that initiate logging, and 
the corresponding information to be logged. Data to be logged may be 
customized for each hypervideo 10,007. For example, a hypervideo 10,007 may 
1 0 create data that logs each hotspot 8003 that is activated when a viewer points and 
clicks on the hotspot 8003. The hypervideo 10,007 may also log each occurrence 
of a pointer being positioned over a specified hotspot. 

A hypervideo scripting language, further described below, may be used to 
define the activities and the corresponding information to be logged during the 
1 5 performance of a hypervideo 1 0,007. 

The logging tools 25,047 may permit application managers to statistically 
analyze and interact with a hypervideo application based upon the logged data. 
So that it may be readily accessed by application programs (s) 25,038, and their 
managers, logged data niay be stored in logging file(s) 25,049 in a standard 
20 format. For example, logged data may be stored in logging file(s) 25,049 in 
database formats supported by the application program(s) 25,038. 

The hypervideo system 25,000 may log data pertaining to a specific 
viewer's interaction with a hypervideo 10,007. Thus, an application 
administrator may be informed of the viewer's interaction with the hypervideo 
25 1 0,007. The hypervideo 1 0,007 may also log data pertaining to specific 

interactive features (e.g. hotspots 8003) actuated by any viewer of the hypervideo 
10,007. 

Authoring Tool GUI for Hvpervideo System 
30 In one embodiment, the authoring tool GUI 26,001 for the hypervideo 

system 25,000, illustrated in Figure 26, is similar to the authoring tool GUIs 
10,001, 15,001, 20,001 described above, but may have additional features. One 
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additional feature may be that the authoring tool permits more windows to be 
displayed simultaneously. Some of the other extra features in the authoring tool 
have their own window. For example, in addition to the Media Warehouse 
window 26,003 and the Workshop window 26,01 1, the authoring tool GUI 
5 26,001 for the hypervideo system 25,000 may include a Queries window and a 
Handlers Notebook window, later described. 

Any window, such as window 'x', may be displayed by actuating a "View 
'x' Window" button on the control bar 26,007 of the authoring tool GUI 26,001 
for the hypervideo system 25,000. The authoring tool GUI 26,001 for the 
10 hypervideo system 25,000 will now be described in further detail. 

Media Warehouse Window 

The Media Warehouse window 26,003 may support media file types 
including: Video, Image, Text, Sound, Music, HTML, Executable, Animation, 
1 5 Virtual Reality Modeling Language (VRML), QuickTime VR (TM), and SQL 
queries used as targets. 

Database Integration 

In one embodiment, a database 25,037 that is object oriented may be used 

20 to enhance the user-friendliness of the authoring tool GUI 26,001 of the 

hypervideo system 25,000. Queries to the database 25,037, or other external 
data sources including applications 25,038, may be objects, which may be 
created, edited, manipulated, and referred during the authoring process. These 
query objects may be used as targets, linked to media objects, that for example 

25 may require a hypervideo's viewer to provide information so that the query object 
may retrieve data from a database 25,037 or other external data source. The 
retrieved data may be used to vary parameters of a hypervideo 10,007. 

Queries Warehouse Window 
30 The authoring tool GUI 26,001 of the hypervideo system 25,000 includes 

a Queries Warehouse 26,072 and corresponding Queries Warehouse window 
26,071 . Query object elements may reside in the Query Warehouse 26,072. 
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Icons 26,073 of Query object elements may reside in the Queries Warehouse 
window 26,071 . The Queries Warehouse may function analogously to the Media 
Warehouse 1003. Thus, the query object parameters defined in a dynamic 
hypervideo may be saved with other dynamic hypervideo parameters, for 
5 example, in a project file or in a database 25,037. 

The Queries Warehouse window 26,071 may function analogously to the 
Media Warehouse window 26,003. The query object elements may be associated 
with query objects. The Query Warehouse window 26,071 may permit the query 
objects to be manipulated. 
10 The Query Warehouse window 26,071 may include a control bar 26,075 

that includes relevant buttons, for example buttons for Properties 26,077, New 
Query 26,079, Detailed List 26,081, etc. The Queries Warehouse window 
26,071 may be displayed or hidden as desired by the authoring tool GUI 26,001 
user. 

15 

Creating a New Query 

In order to create a query object, or to import a query object from another 
file, the New Query button 26,079 may be actuated, such as by pointing and 
clicking. Actuation of the New Query button 26,079 may cause a Query dialog 

20 to be displayed. The query corresponding to the query object may then be 

defined, for example, in a step by step manner, in the Query dialog, or wizard. 
Upon defining the query in the Query dialog, a corresponding query element may 
be placed in the Query Warehouse 26,072, a corresponding icon 26,073 of the 
query element may be placed in the Query Warehouse window 26,071, and a 

25 corresponding query object is created. The query associated with the query 
object may be later edited using the Query Properties page which is later 
described. 

In one embodiment, when accessed, successive pages of the Query dialog 
are displayed to permit the step by step entry of the query. In another 
30 embodiment, the query may be defined with the standard query language (SQL). 
In another embodiment, only common or basic SQL commands may be supported 
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by the Query dialog. In yet another embodiment, more complex SQL queries 
may be implemented by allowing the SQL queries to be textually edited. 

The first displayed page is the Data Source page 27,001 of the Query 
dialog 27,000, illustrated in Figure 27. The source of data 27,003 upon which a 
5 query operates may be defined in the Data Source page 27,001 . In one 
embodiment, any SQL database, having an ODBC driver installed, may be 
selected. Upon specifying a source of data 27,003, the database file upon which 
the query operates may be defined in the Data Source page 27,001 . 

Then, the query may be defined. The query may be a predefined query 
1 0 selected from predefined queries listed on the Query page 27,005. If selected, the 
parameters of the predefined query may be provided to the query object, and may 
be later altered. 

If a predefined query is not chosen, then the query may be defined. To 
define the query, columns 28,005 to be used in the query may be first selected 

15 from available columns 28,005 in the selected database file using the Choose 
Columns page 28,000, illustrated in Figure 28. The Choose Columns page may 
also be used to display tables in the selected database file. 

Upon selecting columns 28,005, the query may be defined in the Filter 
Data page 29,000, illustrated in Figure 29. In the Filter Data page 29,000, the 

20 query may be defined by combining some or all selected columns 28,005 with 
logical operators 29,003. 

The query 30,001 may be reviewed and edited in another page, the SQL 
page 30,000, illustrated in Figure 30. Further, in the SQL page, the Access 
Database option 30,003 may define the frequency of database file access, 

25 permitted by the query 30,00 1 , to extract data from the database file. In one 
embodiment, the database file may be accessed everytime the query 30,001 is 
used so that current database information is provided in response to the query 
30,001. In another embodiment, the database file may be accessed a limited 
number of times, such as just once, for example when the query 30,001 is first 

30 used, or prior to dynamic hypervideo performance. Upon actuating the 

"Advance" button, a more sophisticated query editor may be displayed. In the 
more sophisticated query editor, more complex queries may be created, reviewed 
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and edited. Upon defining a query 30,001, the resulting query 30,001 may be 
used to extract data, that satisfies the query 30,001, from the database file. 

Queries Property Page 
5 The Queries Properties sheet may be accessed, or opened, by actuating, 

such as by pointing and double-clicking on the corresponding query element icon 
26,073, or by selecting the query element icon 26,073 and actuating (e.g. 
clicking) the Properties button 26,077. The Query Properties sheet may include 
general information about a query 30,001, for example, the corresponding SQL 

10 statement, and default parameters which may be applied when using the query 
30,001 as a target 7003 or to modify dynamic hypervideo parameters. 

In one embodiment, the Queries Property sheet may include General, 
SQL, and Display pages. In the General page, the database 25,037 and database 
file, query name, and frequency of database file access may be shown. In the 

1 5 SQL page, the SQL statement may be displayed, an SQL statement editor may be 
provided, and the SQL dialog may be accessed. The Display page may include 
default display parameters that may determine how the results for the 
corresponding query object are displayed to a viewer. In another embodiment, 
the Queries Property sheet may be the same as the Queries dialog. 

20 

Query Objects 

Query objects may be used in different ways for different proposes. In 
one embodiment, a query object may be a target 7003 linked to a media object, 
such as a hotspot 8003 or a notification frame 1157. The corresponding query 

25 30,001 is performed when the media object is actuated. 

In the Media Workshop window 26,01 1 , the query object may be linked 
to a media object, including a notification frame 1 157, by dragging a link icon 
31 ,001 from a Targets page 31,000, illustrated in Figure 3 1, of a media object to 
the Query Warehouse window 26,071 and dropping the link icon on the desired 

30 query element icon 26,073. As may occur when linking other media to a media 
object or notification fi-ame 1 157, a Query Target (or Link) Properties page may 
be displayed. Parameters related to linking the corresponding query element, and 
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media object or notification frame may be defined in the QueryTarget Properties 
page. Parameters relating to how data retrieved by the corresponding query 
30,001 are displayed may also be defined in the Query Target Properties page. 
In another embodiment, the query object may also be used as a marker 
5 1 1 55 for a media object, for example to display the information retrieved by the 
query as a tool-tip for the media object. Thus, when the query 30,001 is 
performed, the information displayed by the marker 1 1 55 may be updated. 

In yet another embodiment, the query object may be used to modify 
dynamic hypervideo parameters. The query object may be a variable in a logical 
10 expression (e.g. Boolean expression). The data retrieved by the corresponding 
query 30,001 may be compared against a constant value or other query objects's 
data. The results of the logical expression may be used to modify dynamic 
hypervideo parameters. The logical expression may be used by a hypervideo 
object's event handler, described below. 

15 

Hvpervideo Application Features 

A dynamic hypervideos may be implemented in an object oriented manner. 
Thus, the dynamic hypervideo parameters, stored in a database 25,037 or project 
file 1670, may be implemented in an object-oriented manner. 
20 A dynamic hypervideo may include the following object types: 

1 , Media element objects corresponding to media files, and having the media 
files's general properties. 

2. Media objects corresponding to hotspots 8003 and notification frames 
1 1 57, as described above. 

25 3. Target (Link) and Target (Link)-level objects, which may be used to 
define respectively targets 7003 such as media objects, and target levels. The 
target and target-level objects may define the actuation properties of targets 
7003. 

4. Query objects, as described above. 

30 



Marking a Media Object 
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In addition to the marking 1 1 55 described above, a media object in a 
dynamic hypervideo may be marked with SQL markers, tool-tip markers, and text 
marking. As described above, the SQL markers may display the results of a 
query, for example in a tool-tip format, and may be proximate to the media 
5 object. Text marking may display text, for example with no background coloring, 
and may be proximate to the media object. Tool-tip marking may display text 
with background coloring, and may be proximate to the media object. 

Target and Target Level Objects 
10 Target objects may also be implemented with SQL queries, or SQL links, 

and with a scripting language, or automatic linking. Target level objects may 
defined all targets 7003 on a level 760 1 . 

Hotframe Objects 

1 5 A hotframe object is a type of media object corresponding to a 

notification frame 1 1 57. Each frame in a video may either be or not be a 
notification frame 1 1 57. Thus, each frame may contain one or zero hotframe 
objects. A hotframe object may be linked to targets 8003 in the same manner as a 
hotspot 8003. The hotframe object may be linked to multiple targets 8003 and 

20 may have multiple link levels. When displaying a frame in which a hotframe 
object was defined, the hotframe targets 8003 may be executed. To create a 
hotframe object, the corresponding frame is selected, and then the Create 
Hotframe button, in the Hotframes Toolbar, is actuated. The Hotframe 
Properties sheet is then displayed so that the hotframe object properties may be 

25 modified and the targets 7003 for the hotframe object defined. Once the 

hotframe object is defined, a hotframe icon 26,501 is displayed in the right-top 
comer of the corresponding frame. The Hotframe Properties sheet for the 
hotframe object linked to the frame may be accessed by actuating, such as 
double-clicking with a left mouse button, the hotframe object, or by clicking on 

30 the hotframe with the right mouse button and selecting properties from a 
displayed list. 
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A Hotframes page may be included in the Media Properties sheet of a 
media element. This Hotframes page may identify all the hotframes defined in 
that media, and display a thumbnail of the corresponding frame in which the 
hotframes are defined. 

5 

Hptft^mes Tpplbar 

In one embodiment, the authoring tool GUI 26,001 of the hypervideo 
system 25,000 includes a Hotframes toolbar. The Hotframes toolbar may be 
similar to the hotspot toolbar and provides similar functions to manipulate 
1 0 hotframe objects. The Hotframes toolbar may be a floating toolbar which can be 
docked to the borders of the Workshop window 26,01 1 . 

One embodiment of a Hotframes toolbar 32,000 is illustrated in Figure 32. 
The Hotframes toolbar 32,000 may include the following buttons: 

Create Hotframe 32,001 - Actuating (e.g. by pointing and clicking) the 
1 5 Create Hotframe button may create a new hotframe object in the current frame. 
If there is already a hotframe object defined in the current frame, the Create 
Hotframe button 32,001 may be disabled. 

Goto Hotframe 32,003 - Actuating (e.g. by pointing and clicking) the 
Goto Hotframe 32,003 button will cause the corresponding frame in which the 
20 hotframe, selected in the Hotframe Combo box, to be displayed in the Workshop 
window 26,01 1. 

Hotframes Combo box 32,005 - The Hotframes Combo box 32,005 
contains the titles of all hotframe objects defined in the media present in the 
Workshop window 26,01 1. 

25 Hotframe Properties 32,007- The Hotframe Properties button may cause 

a Hotframe Properties page, for the hotframe object associated with the frame 
currently displayed in the Workshop window 26,01 1, to appear. 

Remove Hotframe 32,009 -Actuating (e.g. by pointing and clicking) 
Remove Hotfi-ame button 32,009 may delete the hotframe object, and all 

30 associated information, that was defined for the frame currently displayed in the 
Workshop window 26,01 1 . 
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Hotframe Properties Page. 

The Hotframe Properties sheet 33,000, illustrated in Figure 33, may 
include at least two pages, a General page 33,003 and a Target, or Links, page 
33,005. The Target page 33,005 may be identical to the Target page in the 
5 Hotspot Properties sheet. The General page 33,003 includes general information 
about the corresponding hotframe object, including hotframe object name and 
description (which can be modified by the user), and the name of the media with 
which the hotframe object is associated. The General page 33,003 may also 
include an "Enable" check box, which if unchecked may disable the 

10 corresponding hotframe object (so that the corresponding hotframe may not be 
activated) without requiring that the hotframe object be deleted. 

The frame with which the hotframe object is associated can also be 
changed in the General page 33,003. The slider 33,701 or the Next/Previous 
buttons 33,801 may be manipulated to select a new frame with which to associate 

1 5 the hotframe object. Then, the "Change frame" check box may be selected, or 
checked, to associate the hotframe object with the new frame. If the selected 
frame already has a hotframe object associated with the selected frame, then a 
dialog box may be displayed and may include the following options: 
Cancel - Make no changes to selected frame. 

20 Overwrite - Associate the hotframe object lo the selected frame and delete 

the previous hotframe object which was previously associated with the selected 
frame. 

Merge - Merge the two hotframes objects into a single, new hotframe 
object that includes the properties, such as the targets 7003, of the two hotframe 
25 objects. The new hotframe object may be associated with the selected frame. 

Content Design for Large Scale Media Databases 

Problems may arise when randomly accessing large video files during 
performance of a hypervideo 10,007. These problems may be overcome with off- 
30 line auto-tracking, off-line automatic scene-change recognition, and off-line 
automatic linking according to subjects. 
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Stage Management 

To create a hypervideo 10,007, tools other than an authoring tool 1001 
may be needed. In one embodiment, the authoring tool 1001 may only be used to 
create hypervideos 10,007. However, in one embodiment, additional features 
5 may be provided to the authoring tool 1001 so that it can be used to create 
additional functionality for hypervideo applications. Some of the additional 
features may include the ability to define what will happen around (e.g. in the 
background) of the hypervideo 1 0,007, provide menu options from hosting 
applications, and the ability to integrate HTML files. 

10 

Rrnadcastinp and Recording 

A performing dynamic hypervideo may be recorded by actuating a switch. 
The recorded hypervideo performance may then be later displayed offline in a 
linear fashion. 

15 

Dvnamic Hvpervide o Events. Commands, and Handlers 

The hypervideo system 25,000 may support event driven programming. 
Events are actions, including viewer actions, which may be related to a dynamic 
hypervideo. In one embodiment, event driven programming may be implemented 

20 in an object-oriented manner. The hypervideo system's support for event driven 
programming may include an extensive bank of events and commands, and the 
ability to define event handlers, to receive event messages, with the authoring tool 
GUI 26,001 of the hypervideo system 25,000. 

As with the run-time module 1001 (or any OLE object), communication 

25 between the run-time module 1001 and the surrounding environment is achieved 
by using events and commands. An object associated with the dynamic 
hypervideo, during the performance of a dynamic hypervideo, may produce and 
fire an event message that indicates that a certain event occurred with respect to 
the object. Hypervideo applications may communicate with application 

30 program(s) 25,038 by firing event messages to the application program(s) 25,038. 
A recipient application program 25,038 may be affected upon receiving and 
processing an event message. Further, event messages fired by a dynamic 
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hypervideo make the components of the hypervideo system 25,000 that receive 
the event messages aware of the dynamic hypervideo' s status. 

Commands may be sent by an application 25,038, or by an event handler 
in a hypervideo 10,007 to manipulate the performance of the same or another 
5 hypervideo 1 0,007. For example, the commands may alter hypervideo 10,007 
interactive features or other parameters, or may initiate the performance of a 
media within the hypervideo 10,007. Commands may be sent to a hypervideo 
1 0,007 as a result of the user interaction with another hypervideo 1 0,007, or in 
response to an event triggering the command from an event handler. 

10 Objects in a hypervideo 10,007 may include media objects, including 

objects for hotspots, and objects for media, targets, target-levels, and frames. 
Each class, that instantiates an object in a hypervideo 10,007, may include a 
method permitting the object to fire an event message. Further, each object may 
be defined during hypervideo 10,007 authoring to have a method(s) that are event 

1 5 handler(s) for receiving event message(s) during hypervideo performance. An 
event handler is a method of an object that defines how the object responds to a 
received event message. Even if a hypervideo 10,007 may not communicate 
externally, objects of the hypervideo 10,007 objects may still communicate with 
one another by event messages. 

20 The information retrieved from queries or logged information may be 

used to manipulate dynamic hypervideo performance. Event handlers may utilize 
variables (e.g. placeholders) corresponding to information resulting from queries, 
or the retrieval of logged information not performed by queries. An event 
handler may utilize such information to manipulate dynamic hypervideo 

25 performance. Both queries and event handlers may be executed by the dynamic 
hypervideo server 25,035. 

Objects and Events 

Numerous classes of objects may exist in a hypervideo 1 0,007. Many 
30 objects corresponding to each class may be instantiated. Each class of objects 
may have a unique set of event handlers associated with the class. Not all event 
handlers, which are methods, may be found in each class because some event 
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handlers are not relevant for certain classes. For each and every object in a 
hypervideo 10,007, events in an object type's event list may be selected for 
handling, either by using a handler specified for that specific event, or by firing 
the events to be handled in an external environment. 



5 An exemplary list of object types and their associated events for one 

embodiment follows: 



Object Type 


Event 


Description 


Protect 


OnStart 


The nroiect started 




OnEnd 


The project ended 

normallv 




OnResize 


The project window was 
resized. 




fin' 1 Ar^tnmA^ A 


1 ne projeci enacQ uue lu 
external interference. 


noiDpoib 


On \4 rfci 1 c** 1i W 
V/I J 1 VI U UoC 1 lu J\ 


1 IJC ilV^loUUl Woo CIL^lLIClldJ 

by clicking the left mouse 
button. 




OnMouseOver 


The cursor was passed 

over the hotspot. 




OnMouseRightClick 


The hotspot was actuated 
by clicking on the right 
mouse button. 




OnStart 


The hotspot appeared for 
the first time. 




OnEnd 


The hotspot no longer 
exists in the hypervideo. 


Hotframes 


OnStart 


The hotframes 's frame 
display started. 




OnEnd 


The hotframe's frame 
display ended. 
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Media 


OnStart 


The media started 
playing. 


OnPause 


The media ended playing 
normally. 


OnStop 


The media ended playing 
due to external 
interference. 


LinkLevel 


OnStart 


The link (target) level 
started playing. 


OnEnd 


The link (target)level 
ended playing. 




OnTerminate 


The link (target) level 
ended playing due to 
external interference. 


Link 


OnStart 


The link (target) started 
playing. 




OnEnd 


The link (target)ended 

playing. 




OnTerminate 


The link (target) ended 
playing due to external 
interference. 


Frame 


OnStart 


The frame display 

started. 




OnEnd 


The frame display ended. 



When an event message, corresponding to an event, is fired from the 
15 client 25,003 to, for example, to the dynamic hypervideo server 25,035, the event 
message may include arguments, or parameters that indicate the object that has 
fired the event message. An argument may include a string specifying the name 
of the object. 
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Each object may have a Properties sheet 34,001, illustrated in Figure 34, 
that includes an Events page 34,003. In the Events page, the events 34,005 
whose messages may be fired by the object may be specified (e.g. selected, by 
checking a box, from all events potentially fired by the corresponding object). 
5 From the Events page 34,003, event handlers, corresponding to the specified 
fired events may also be accessed by actuating (e.g. clicking) a Handler button 
34,007, proximate to each specified event. 

The Events page 34,003 may also include a Select All button 34,009, 
which when actuated, selects all events potentially fired by the corresponding 
10 object. The Events page 34,003 may also include a Set As Default button 34,01 1 
which makes the selected events in the currently displayed Events page 34,003 
the default configuration for the' Events pages 34,003 for other objects of the 
same type. 

In one embodiment, the Events page for an object is located in the 
1 5 properties sheet for that object. For example, the Events pages for the following 
objects are located in the co-listed properties pages, or as otherwise described 
below: 

Project object- In the Project Setting Properties sheet. 
Hotspot object - In the Hotspot Properties sheet. 
20 Hotframes object - In the Hotframe Properties sheet. 

Media object - In the Media Properties sheet. 

Target Level object - In the links, or target, page in a Hotspot Properties 
sheet. Each level tab, or page, may include an Events button which may be 
actuated to display, or pop-up the corresponding Link Level object events sheet. 
25 Target, or Link, Object - In the Media link properties sheet. 

Frame Object - The Workshop toolbar may include an Events button 
which will display, or pop-up, the Frame object events page for 
the current frame. 

30 Commands 

Hypervideo application commands may enable the performance of a 
hypervideo 10,007 to be manipulated. The commands may be issued by 
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application(s) 25,038 or by the player 25,039, for example in response to the 
occurrence of an event. The commands may, for example, manipulate the 
interactive features of the hypervideo 10,007 by enabling or disabling hotspots 
8003, manipulate the properties of different objects in the hypervideo 10,007, or 
5 launching a target 7003. Commands may include parameters which identify the 
action to be performed and the object to which the action is related. In one 
embodiment, the commands that may be submitted to a hypervideo 10,007 
include: 



Commands 


Parameters 


Description 


PlayO 






Pause() 






CloseQ 






EnableHotspotO 


1 . Boolean Enable Switch 

2. String with Media Name 

3. String with Hotspot 
Name 


Enable or disable the 
specified hotspot in the 
specified media. 


ActivateMediaQ 


1 . String with Media Name 




ActivateLinKO 


1 . String with Lmk (target) 
Name 


Launch the specified 
Link (target) level. 
This command will 
launch only the 
specified level and not 
the entire set of link 
(target) levels 
associated with a media 
object. 


ActivateHotspotO 


1. String with Media Name 




SetProjectPropertiesO 






SetHotspotPropertiesQ 
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SelMediaPropertiesQ 






SetLinkPropertiesQ 






SetLinkLevelPropertiesQ 







5 



These commands may be supported by the dynamic hypervideo server 25,035 and 
player 25,039. 

Event Handlers 

10 An event handler is a script associated with a specific event message that 

may be received by a specific object. Event handlers enable relatively 
inexperienced hypervideo 10,007 authors to create relatively complex 
hypervideos 10,007, for example using substantially all supported events and 
commands, and to permit communications between application programs 25,038 

1 5 and other components of the hypervideo system 25,000. 

General 

In one embodiment, each object in a hypervideo 10,007 may have a 
unique event handler associated with each event message that the object may fire. 

20 Each event handler also may access parameters, for example object name, of the 
object that fired the event. These object parameters may be needed by the event 
handler to process the received event message. 

In one embodiment, event handlers may be written in a simple scripting 
language, having the following form: 

25 On Event {Condition}Do Command 

Event handlers 26,203 may be viewed in the Handlers Notebook window 26,201, 
illustrated in Figures 26 and 35, in the authoring tool GUI 26,001 , or through an 
Events page. In the Handlers Notebook window 26,201 , different pages 26,209 
displaying different event handlers 26,203 (each associated with an unique event 

30 message that may be fired by an unique object) that may be reviewed, edited or 
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created. A page 26,209 from the Handlers Notebook window 26,201 may also 
be displayed in the Events page, discussed above. 



Handlers Notebook Window 
5 The Handlers Notebook window 26,201 may be hidden in the authoring 

tool GUI 26,001 . In one embodiment, the Handlers Notebook window 26,201 
includes pages each of which includes a unique event handler 26,203 definition. 
Blank pages may be opened, and new handlers may be authored therein. New 
handlers created through an object's Events page are automatically added to new 

1 0 pages in the Handlers Notebook window 26,201 . 

In one embodiment, a particular page 26,209 which may contain a 
particular event handler 26,203 may be accessed in one of two ways. First, 
regardless as to whether an event handler is defined in it or not, a page 26,209 
may be selected with specific event handler 26,203 parameters (e.g. object type 

1 5 and name, and the specific event) from combo boxes 26,205 in the Handlers 
Notebook window 26,201 . If the selected page includes a predefined event 
handler, the event handler 26,203 may be displayed in the page 26,205 in an area 
26,21 1 for displaying script. If the selected page does not include a predefined 
event handler, the script display area 26,21 1 may be blank. Pre-defined handlers 

20 on pages may be selected by manipulating (e.g. clicking) the Next and Previous 
Handler buttons 26,213. A new event may be created, or a pre-existing event 
edited in the script wizard, described below, which can be launched by actuating 
(e.g. click) the script wizard button 26,21 7. 

25 Handler's Scripting Language 

In one embodiment, the script associated with an event handler 26,203 
may be defined in a scripting wizard. In one embodiment, the script area 26,21 1 
in the Handler notebook, may be used to conveniently present handlers described 
in script form to users who are comfortable with the concept of scripting. Other 

30 users may review the event handler script to determine whether a handler has 
been created, and the complexity of the event handler. In this embodiment, any 
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modifications made to the event handler may be performed with the script wizard, 
in addition to altering the text in the text editing window. 

In another embodiment, a more complicated and powerful scripting 
language (e.g. VisualBasic or JavaScript) may be used to permit experienced 
5 script programmers to integrate the application(s) 25,038 with other hypervideo 
system 25,000 components using events and commands. 

The structure of one embodiment of an event handler follows: 

Ai ObjectType ObjectName 

On EventName 

10 {If (QueryConditionl \ LoggingConditionl) }Commandl 

{If (QueryCondition2 \ LoggingConditionl) } Command! 

End 
For Example: 

At Hotspot Mosque 
1 5 OnMouseClick 

EnableHotspot (FALSE, Jerusalem, Church) 

If (QueryOOl equal 10) ActivateLink (LinkMuslimOOl) 

End 

Note that data from a query or a log file 25,049 may define, at least in part, a 
20 logical expression, or conditional term, of an event handler 26,203. 

Each event handler 26,203 may contain an unlimited number of 
commands, some of which have been previously discussed. Commands may be 
used to manipulate hypervideo 10,007 parameters. Each command may be 
placed on a separate line. Each command may be accompanied with a logical 
25 expression. The logical expression may include a placeholder, or variable, 
associated with a query or logged information. The command may be 
implemented only if the logical expression is satisfied. The logical expression 
may be a simple Boolean expression which compares the value of a first variable 
to the value of a second variable or constant. 

30 

Script Wizard 
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The Script Wizard 36,000, whose window 36,001 is illustrated in Figure 
36, is used to generate the script for event handlers 26,203. The Script Wizard 
36,000 permits an event handler 26,203 to be visually programmed without 
manual typing (e.g. code is generated based upon user selections in the Script 
5 Wizard 36,000). The Script Wizard window 36,001 may be launched from the 
Handlers Notebook window 26,201, by actuating (e.g. clicking) the Script 
Wizard button 26,217. 

In one embodiment, scripts may be defined with the Script Wizard in two 
steps. First, commands and associated parameters may be defined. Second, 

1 0 conditional terms may be defined to manipulate hypervideo 1 0,007 parameters, 
described above. When a command is defined, the user can either choose the 
desired command from the combo-box, or click the "Alphabetical Command 
List" button or "Topical Command List" button, and choose the command from a 
pop-up menu. Upon command definition, a dialog may be displayed (e.g. popped 

1 5 up). Parameters associated with the command may be defined in the dialog. The 
dialog may also be displayed, permitting the parameters to entered or edited, by 
actuating (e.g. clicking) the Modify Parameters button 36,003 of the Script 
Wizard window 36,001 . The defined parameter may be displayed in the 
parameters line 36,005 of the Script Wizard window 36,001 . 

20 Logical expressions may be added to the command (so that the command 

may be performed only if the logical expression is fulfilled) upon selecting (e.g. 
checking) the Add Expression box 36,007. Operands, including variables, and 
operators for the conditional terms may then be selected. The operators may be 
selected from combo box having a list of operators (including equal, greater than, 

25 etc.). To select an operand, a button may be actuated to display menu 37,001 , 
illustrated in Figure 37, which contains a constant 37,003, a query 37,005 (from 
the Query Warehouse window 26,072), or a logging variable 37,007. Selected 
parameters are displayed in the appropriate field, and can be modified at any time 
in the Script Wizard window 36,001 . Definition of the script in the Script Wizard 

30 window is completed by actuating (e.g. clicking) the OK button 37,009. Then, 
the script can be viewed in the Handlers Notebook Window 26,201 . 
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Multiple sets of commands and logical expressions may be created using a 
multi-level approach, similar to the link, or target, levels described above. The 
commands sets may be manipulated, for example by dragging and dropping the 
commands to pages, or tabs, corresponding to the different command set levels. 

5 

Defining Default Events Settings and Event Handlers 

Default settings may be defined for each object type. The default settings 
may include the default events configuration for each object type and default 
handlers for each event associated with a specific object type. Upon selecting the 

10 Set Default Settings option in the Settings menu in the main tool and menu bars, 
a Default Settings sheet 38,001, illustrated in Figure 38, may be displayed (e.g. 
popped up). In the Default Settings sheet, each object type may have an Events 
page (e.g. Project Events page 38,003, Hotspot Events page 38,005, Media 
Events page 38,007, Link Events page 38,009, Frame Events page 38,01 1, etc.) 

1 5 which may be identical to the Events page that each object of that type has in its 
corresponding properties page. The Events Settings and handlers defined in these 
Events pages are the default settings for the corresponding object types. Thus, 
any object, of an object type, created after the default settings have been defined 
may have the same selection of events, and corresponding event handlers, unless 

20 otherwise changed. 

The hypervideo system 25,000 may permit hypervideo 10,007 activities to 
be logged during hypervideo 10,007 performance. In one embodiment, an author 
25 of a hypervideo 1 0,007 may specify, during hypervideo 10,007 authoring, which 
viewer interactions (or other hypervideo 10,007 activities) and corresponding 
information are to be logged. The logged information may be used to manipulate 
the performance of the hypervideo 10,007. 

30 General 

In one embodiment, the logging of hypervideo 10,007 activities and 
corresponding information may be performed in relation to objects defined in the 
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hypervideo 10,007. For each object in the hypervideo 10,007, corresponding 
actions to be logged may be defined. In one embodiment, media files may be 
objects, and thus the activities (e.g. starting and stopping) of corresponding 
media files may be logged. This approach may be analogous to the approach 
5 used for defining objects's events. 

In one embodiment, each object may be associated with one of three 
logging levels. Successive logging levels log more extensive information that 
previous types, and may include: 

a. Count: Logs the number of times the object was acted upon (e.g. 
1 0 actuated, performed, stopped, etc. For example, the number of 

times a hotspot was actuated may be logged. 

b. Duration: Beside logging the information associated with the 
Count logging level, the Duration level also logs the duration of 
an object's activity (e.g. performance time of a media file) may be 

15 logged. 

c. Timing: Besides logging the information associated with the 
Duration logging level, the Timing level logs the time when the 
object activities occurred (e.g. when the media file performance 
started and stopped, or when a hotspot 8003 was actuated). 

20 A list of the objects for which related activities may be logged, and the types of 
activities that may be logged for those objects follows: 



Object Type 


Logging Types 
Available 


Description 


Project 


Count 


Number of times a 
hypervideo 10,007 was 
performed, or activated, 
and the identifications 
(IDs) of the viewers who 
activated the hypervideo 
10,007. 
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Object Type 


Logging Types 
Available 


Description 




Duration 


The total time of each 
performance of the 
hypervideo 10,007. 




Timing 


The start and stop times 
of each performance of 
the hypervideo 10,007. 


Hotspots 


Mouse Click : Count 


Number of times a 
hotspot was actuated 
(e.g. clicked) during a 
performance. 




Mouse Click : Timing 


Times when the hotspot 
actuations occurred. 




Mouse Over : Count 


Number of times a 
pointer was positioned 
over a hotspot. 




Mouse Over : Duration 


Total time that a pointer 
was positioned over a 
hotspot. 


Media 


Count 


Number of times a media 
was performed, 
regardless of associated 
link (target), during a 
performance of a 
hypervideo 10,007. 




Duration 


Number of frames of the 
media displayed (e.g. 
during each media 
performance). 
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Object Type 


Logging Types 
Available 


Description 




Timing 


Start and stop times 
(relative to hypervideo 
10,007 start time) of the 
media. 


Link Level 


Count 


Number of times that a 
link (target) level was 
activated during the 
performance of a 
hypervideo 10,007. 


Duration 


Activation time duration 
of a level. 


Link 


Count 


Number of times a target 
was executed during the 
performance of a 
hypervideo 10,007. 


Duration 


Time duration of target 
execution. 


Frame 


Count 


The number of times a 
frame was displayed. 



10 Logging Design 

In one embodiment, a logging tier, or level, may be selected for each 
object. Each object may have a Logging page 39,001 , illustrated in Figure 39, in 
its properties sheet. In the Logging page 39,001, the logging tier may be 
specified. 

1 5 In one embodiment, logging pages 39,001 may be accessed, as described 

below, for the following object types: 
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Project object- In the Project Setting Properties sheet. 
Hotspot object - In the Hotspot Properties sheet. 
Hotframes object - In the Hotframe Properties sheet. 
Media object - In the Media Properties sheet. 
5 Target, or Link, Level object - In the Target, or links, page in the Hotspot 

Properties sheet, each Target Level page will include an Events button 
which when actuated displays (e.g. pops-up) the Target Level Object 

Events 

page from which the Logging page may be accessed. 
1 0 Link Object - In the Media link (target) properties page. 

Frame Object - The Workshop toolbar may include an Events button 
which may display (e.g. pop-up) a Frame object events page for the current 
frame from which the Logging page may be accessed. 



15 Defining Default Logging Setting 

Default logging settings may be defined in a manner similar to the way 
that default settings for the events are defined as described above. Thus, in one 
embodiment, the Logging page 39,001 may include a Set As Default button 
39,002, which may be actuated to make the logging data in the Logging page 

20 39,001 the default data for Logging pages 39,001 for other objects of the same 
type (e.g. any object of the same type that is later created will have the same 
logging data, unless otherwise defined). 

Using Logging Information in a Dvnamic Hvpervideo 

25 In one embodiment, as described above, logged information may be used 

to manipulate hypervideo 10,007 parameters during the performance of the 
hypervideo 10,007, by embedding the logged information (e.g. parameters) inside 
logical expressions as operands, or variables, or placeholders. In one 
embodiment, the operands (e.g. of the logical expressions) for logged information 

30 may be dynamically updated in response to viewer interactions during hypervideo 
10,007 performance. To reference these variables, each variable corresponding 
to a logged parameter may be automatically assigned a logical name by the 
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hypervideo system 25,000. In one embodiment, the logical names may have the 
following format : 

ObjectType_ObjectName_LoggedFeature 

5 An example of a logical name may be: Hotspot_Hotspot001_Count or: 
Media_Jerusalem_DurationLast. A logged parameter may be selected as a 
variable in a logical expression by selecting the logging option in the Query 
Wizard when prompted to choose an operand. Then, a variable corresponding to 
a desired logged parameter may be selected from a combo box. 

10 

Logging during Hypervideo Performance 

A new record of logged information may be created upon the occurrence 
of an event (e.g. the actuation of a hotspot 8003). Each record of logged 
information includes the information, about a specific event, specified by a 

15 hypervideo developer. In one embodiment, logged information record formats 
may include: MouseClick(user_id, project^name, hotspot_name) and 
MediaPlay(userJd, project_name, media^name, duration). In one embodiment, 
the logged information may be stored, by the dynamic hypervideo server 25,035, 
in a file (e.g. log file 25,049 or database), in the form of list, having a proprietary 

20 format. In another embodiment, the hypervideo dynamic server 25,035 may 

facilitate exporting (e.g. with ODBC drivers) and storing logged information in a 
database 25,037, such as a SQL database. Logged information in a database 
25,037 may be viewed and analyzed with the logging tools 25,047, or with other 
database tools. 

25 The logged information may be stored in a client 25,003a so as not to 

diminish the bandwidth available on the network 25,050a for video transmission. 
The logged information may be periodically transferred to the server 25,005 for 
storage during hypervideo 10,007 performance, or upon hypervideo 10,007 
termination. 



30 
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Logging Analysis 

The hypervideo system 25,000 may include logging tools 25,047 which 
may permit viewing the logged infomiation associated with the performance of a 
hypervideo 10,007. The logging tools 25,047 may support retrieving logged 
5 information, including different sets of logged information (e.g. different sets 
corresponding to different performances of the same hypervideo 10,007), to 
permit statistically analyzing the sets of logged information. The logging tools 
25,047 may read and analyze logged information stored in different logging file 
formats (e.g. proprietary file format or conventional database format). 

10 

Hypervideo Application Integration Requirements 

The authoring tool may permit an author to access features and 
functionality in the hypervideo system 25,000. The player 25,039 may include a 
modified version of the run-time module 1001. The player 25,039, which may 
15 include a run-time module 1001, may be integi:ated with the relevant site building 
tools used to build web sites and the resulting web sites. Thus, the player 25,039, 
may be included in a web browser as a standard control (e.g. ActiveX control). 
The player may have to implement interfaces of relevant API architectures (e.g. 
COM) to communicate with the web browser. 

20 

2.5 Object Oriented Implementation of Hypervideo System 
Authoring Tool 

As described above, the authoring tool 1001 facilitates the creation of 
25 hypervideos 1 0,007. One embodiment of a procedure for creating a hypervideo 
10,007, depicted in Figure 40, is described below. 

Author-selected media files may be imported into the Media Warehouse 
window 10,003 (step 40,001). Media elements 1690, that are to be base target(s) 
1 109 for the hypervideo 10,007 may be selected using the project settings dialog 
30 (step 40,002). A media element 1 690 may be dragged into the Workshop 
window 8001 (Step 40,003). A media object may be defined in the media 
associated with the media element 1690 (step 40,004). In one embodiment, the 
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media object is a hotspot, for example, which may be defined in a frame of video 
over a region of interest. The media object may be defined in the frame in which 
the region of interest first appears in the video. Then, the defined media object 
corresponding to the region of interest is tracked in successive video frames 
5 (steps 40,005a,b,c). The media object may be tracked automatically 40,005a, 
manually (or semi-automatically) 40,005b, or by interpolation 40,005c, which are 
described above. In one embodiment, for example, target(s) 7003 may be linked 
to the media object using the Target page 7001 of the Hotspot Properties sheet 
7101 (step 40,006). The shape, or type, of the over and click cursors, described 

1 0 above, associated with the media object may be defined (step 40,007) in the 
Hotspot Properties sheet. A marker 1 1 55 to identify the media object during 
hypervideo 10,007 performance may also be defined (step 40,008). Steps 40,003 
through 40,008 may be repeated to define more media objects in the same or 
different media. Upon completing media object definition, the hypervideo 1 0,007 

1 5 may be saved (step 40,009). In one embodiment, the hypervideo 1 0,007 may be 
previewed (step 40,010) in the authoring tool 1001 or in the run-time module 
1001 . The hypervideo 10,007 may then be published so that is accessible for use 
by run-time modules over computer networks (step 40,01 1). 

In one embodiment, the authoring tool 1001 may be used in conjunction 

20 with other video editing tools, such as Adobe Premiere (Adobe Systems, Inc., 
San Jose, CA). In another embodiment, data defining a hypervideo 10,007 may 
be created and saved in a project file 1670 (e.g. .OBV file) by the authoring tool 
1001 using the following libraries and dynamic linked libraries (DLLs): 

25 a) Object database 1 127 may be a library configured to manage the 

data of the Project File 1670. 
b) MediaDisplay is a library that provides the services 

necessary to display or execute various 
media. 

30 c) FrameManager is a library that provides the services needed 

for extracting fi-ames from within media 
files. 
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d) Tracker 



10 



e) BitMaster 



f) Shapelterator 



g) MCIVA 



is a library that tracks the geometry, size 
and position of a media object in successive 
video frames. 

is a DLL that performs the mathematical 

calculations and data processing necessary 

for the Tracker operation. 

is library that iterates the geometric shape 

of a hotspot into scan-lines for use by the 

Tracker as well as the marking 1155 

mechanism of the run-time module. 

is a run-time module DLL that performs the 

preview function when the authoring tool 

1001 is used. 



15 

In one embodiment, the authoring tool 1001 may be a Microsoft Foundation 
Class (MFC) application. In another embodiment, the authoring tool 1001 may 
also be an Multiple Document Interface (MDI) application. As an MDI, the 
authoring tool 1001 may contain multiple templates (e.g. windows), including the 
20 Media Warehouse window 1 0,003, the Workshop window 8001 , and the Preview 
window 8101. Each template may include document, view, and frame classes. 
This structure of windows and classes is known to persons skilled in the art of 
Microsoft Windows computer programming. The templates and their classes are 
described below: 

25 

Media Warehouse Window 

In one embodiment, the Media Warehouse window 10,003 may be 
implemented with the logical class relationships illustrated in Figure 41 . Within 
the Media Warehouse window 10,003 are the document, view, and frame classes, 
30 described below: 
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may be described by class EObviousDoc 41,003 which may 
include a parameter for a pointer to an EProject object 41 ,002. 

The EObviousDoc 41,003 class may include methods for saving 
hypervideo 10,007 data to and accessing hypervideo 10,007 data 
from a project file 1670 (e.g. respectively OnOpenDocument() and 
OnSaveDocumentQ methods). In one embodiment, the 
EObviousDoc 41,003 may include functions similar to those 
available within EProject 41,002, such as LoadQ and StoreQ for 
respectively accessing and saving hypervideo 10,007 data. 

In one embodiment, prior to performing saving function, the 
hypervideo 10,007 data may be compressed, or encrypted. 
Because the hypervideo 10,007 data may be compressed, the 
document may not be serialized in a standard fashion. 
Serialization of the hypervideo 1 0,007 data may be avoided, for 
example, by overriding the EDocument=sOnOpenDocument() and 
OnSaveDocumentQ methods in the EDocument class. 

may be described by class EObviousView 41 ,005. The 
EObviousView class 41 ,005 may include methods for displaying 
the Media Warehouse window 10,003. The media element 4007 
may be inserted into the view after the project is read and the view 
may have been initialized. Initializing a view class object is known 
to persons skilled in the art of Windows Programming. If the class 
is unable to locate a media file corresponding to a media element 
4007, a help window is opened. The window prompts the author 
to enter data to assist in locating the corresponding media file. 
Upon failing to locate a media file, the corresponding media 
element is deleted from the hypervideo. A similar technique is 
used for cursors 71 1 1, 71 13 in a hypervideo 10,007. 
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A media display function in the EObviousView class 41,005 may 
facilitate the display of thumbnails 4007 corresponding to the 
media associated with the Media Warehouse window 10,003. The 
EObviousView class 41 ,005 may include parameters for storing an 
5 array of thumbnails 4007. The stored thumbnails 4007 may be 

used by other classes in the authoring tool 1 001 . 



In one embodiment, the EObviousView class 41,005 may include 
other methods. In another embodiment, the EObviousView class 

10 41 ,005 may include methods for accessing media properties by 

using methods from the EMedia class which is described in the 
Object database 1 127 section below. For example, the 
EObviousView class 41 ,005 may open a media menu. 
Furthermore, The EObviousView class 41 ,005 may permit the 

1 5 hypervideo 1 0,007 to be previewed by using an application 

program interface (API) of the run-time module 1101, which may 
be a DLL. 



frame: may be described by the EObviousFrame object 4 1 ,004. The 

20 EobviousFrame object 41,004 creates and displays a control bar, 

and menu commands, of the Media Warehouse window 10,003. 



Media is displayed in the Media Warehouse window 10,003 using methods in the 
EMediaList class 41 ,006 . The EMediaList class 41 ,006 may be derived from the 
25 Microsoft (TM) Foundation Class (MFC) CListCtrl which is an object oriented 
programming class known to persons skilled in the art. In one embodiment, in 
addition to displaying media, the EMediaList class 41 ,006 may include additional 
methods, including those that: 



30 a) 



Control cursor appearance and configuration. For example, in one 
embodiment, the EMediaList class 41,006 may include a method that 
permits modifying the cursor type based upon the image behind the 
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pointer. In one embodiment, for example, a "hand-shaped" cursor may be 
displayed when the pointer is positioned over a media. An "arrow" 
cursor may be displayed when the pointer is not over media. 

5 b) Open the Media Properties Sheet 5003. In one embodiment, the Media 
Properties sheet 5003 may be opened when a switch is actuated (e.g. a 
mouse button is double-clicked) while the pointer is positioned over the 
corresponding media element thumbnail 4007. In one embodiment, 
actuation of the ENTER key on the keyboard may open the Media 
10 Properties sheet 5003 when the pointer is positioned over the 

corresponding media thumbnail 4007 and a switch was actuated (e.g. a 
mouse button is clicked). 

c) Manage the display and content of various Windows messages (which are 

1 5 known to persons skilled in the art of Windows Programming) associated 

with functions. In one embodiment, the EMediaList 41,006 class may 
include a method that generates and displays context-sensitive "drag and 
drop" messages for functions such as linking a media object to a target 
7003. In one embodiment, a portion of the Media Warehouse window 

20 1 0,003 may display a hotspot menu. This hotspot menu may be derived 

from a general purpose Menu class in the MFC library and is known in the 
art as a pop-up menu. In another embodiment, the hotspot menu is 
opened when a switch is actuated (e.g. the alternate mouse button is 
clicked) while the pointer is positioned over the corresponding media 

25 element thumbnail 4007. In one embodiment, each item in the hotspot 

menu may be the name of a hotspot 8003. In another embodiment, the 
menu items may include thumbnails 4007 of the corresponding hotspots 
8003. In one embodiment, upon selecting a hotspot menu item, targets 
7003 of the selected hotspot 8003 are displayed. Targets 7003 are 

30 displayed by popping up another pop-up menu with a list the selected 

hotspot's 8003 targets 7003. In this Targets 7003 pop-up menu, the 
targets 7003 may be identified by name. In another embodiment, a 
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thumbnail 4007 of the media element associated with the target 7003 may 
be displayed. 

Workshop Window 

5 In one embodiment, a hypervideo 1 0,007 may be created in the Workshop 

window 8001. In a Microsoft Windows (TM) environment, within the Workshop 
window 8001 are the document, view, and frame classes, described below: 

document: may be described by class EClipDoc 42,003, depicted in Figure 
1 0 42. In one embodiment, the class EClipDoc 42,003 may include 

parameters for storing data associated with the current frame 
being displayed in the Workshop window 8001, as well as a list of 
objects instantiated from a class EDrawObj 42,002. In another 
embodiment, the class EClipDoc 42,003 may also include 
1 5 parameters for storing hotspot tracking 1 029 and other data. In 

one embodiment, the class EClipDoc 42,003 may include methods 
for displaying a video frame or an image in the Workshop window 
8001, and for providing hotspots 8003 associated with (e.g. on 
top of) the video frame or image displayed in the Workshop 
20 window 8001. 

view: may be described by class EClipView 43,002, depicted in Figure 

43, may include methods, including "Draw Rectangle", "Cut" and 
"Undo". In one embodiment, the class EClipView 43,00 also may 

25 include methods for creating, editing, and manipulating hotspots 

8003. The class EClipView 43,002 further may include a method 
for displaying a window in which a video frame or an image is 
displayed. In one embodiment, the window is located inside the 
EClipView window, and in another embodiment, the window may 

30 be smaller than the screen area and projected as a function of the 

size of the media and author-defined specifications. These 
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specifications, for may be defined in the Options Settings menu 
item in the main menu bar of the authoring tool 1001 . 

frame: may be described by class EClipFrameWnd 44,002, depicted in 

Figure 44, and may include, in one embodiment, methods for 
5 creating and manipulating the workshop toolbar, the hotspot 

toolbar 9001, and the workshop control bar illustrated with 
buttons in Figure 8b, and corresponding functions (e.g. tool 
functions). 



The following tool functions may be available from within the Workshop window 
10 8001: 
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Draw Objects 

In one embodiment, the class EDrawObj 42,002 and its derived classes 
may include methods for managing hotspot 8003 data associated with a 
displayed frame. In another embodiment, for a frame-object, instantiated 
5 from class EFrameObject 42,005, corresponding to the currently 

displayed video frame, the authoring tool 1001 may create an object 
instantiated from class EDrawObj 42,002. In addition to including a 
parameter for storing a pointer to a frame-object, class EDrawObj 42,002 
may include methods for drawing, resizing, and moving a hotspot 8003. 

1 0 Classes which may be derived from class EDrawObj 42,002 include: 

EDrawRect 42,006 which may include methods for drawing 
non-polygon-shaped hotspots 

1 5 EDrawPoly 42,007 which may include methods for drawing for 

polygon-shaped hotspots. 

In one embodiment, an object is instantiated from the class EDrawObj 42,002 
when a new hotspot 8003 is drawn. In another embodiment, when creating the 

20 EDrawObj 42,002 object a new ECIipObject 43,01 1 is instantiated for the video, 
or clip, (e.g. in an object database 1 127) as well as a new EFrameObject 48,021 
with the appropriate geometry. In one embodiment, the hotspot name may be 
assigned automatically and may feature a consecutively numbered suffix. The 
object oriented representation of a hypervideo 10,007 project is further described 

25 below. In one embodiment, the EDrawObj may handle the visual representation 
of a hotspot 8003 while the ECIipObject 48,026 and EFrameObject 48,021 
objects handle the object database 1 127 representation of a hotspot 8003. 



wo 99/10822 



PCT/US98/I7444 



158 

Draw Tools 

In one embodiment, the draw-tools are the classes that perform operations 
in the workshop 8001 . The author selects a draw-tool by choosing from 
the options displayed on the hotspot tools toolbar 9001 . The current 
5 draw-tool may be stored in the static member EDrawTool::c_drawShape 

and, in one embodiment, options include: selection, setjarget, reel, 
triangle, ellipse and poly. In one embodiment, the draw tools are also 
responsible for setting the cursor to the appropriate configuration, 
namely, pencil, arrow or set target cursors. 

10 

The draw-tool classes (derived from EDrawTool 43,003) may include: 



ESelectTool 43,006 In one embodiment, this may be the default tool. 

In one embodiment, the author can move the 

1 5 handle by actuating a switch (e.g. clicking on the 

left mouse button) while the pointer is positioned 
on the handle of a selected hotspot 8003. In an 
object oriented sliape drawing tool, the may handle 
are areas often positioned at vertices of a shape, 

20 which an author can move in a two dimensional 

space and by moving the handle, and resizing the 
drawn shape. In one embodiment, the hotspot 8003 
may be selected if the cursor was positioned within 
the fi-ame of a defined hotspot 8003. In one 

25 embodiment, multiple hotspot 8003 selections may 

be made by depressing the shift key while 
manipulating the mouse and its buttons. 

In one embodiment, all hotspots located within a 
30 rectangular region may be selected using a net 

selection. A net selection is a term known to 
persons skilled in the art. A net selection may begin 
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when drawing of a rectangle by depressing and 
holding down the left mouse button. In one 
embodiment, the hotspot(s) selected are stored in 
the global variable selectMode. The variable may 
reflect a value of none (meaning no selection was 
made), netSelect^ move (meaning that a hotspot 
8003 may have been dragged to a new location), 
size (a hotspot is being resized) and to_bej:reated 
(a hotspot 8003 is being created - wherein this 
mode is set by draw tools 43,003 that creates 
hotspots 8003). 

In one embodiment, this tool may be used for 
drawing non-polygon-shaped hotspots 8003. One 
method of defining non-polygon-shaped hotspots 
8003 will now be described. In one embodiment, 
the hotspot may be defined by a bounding 
rectangle. Clicking the left mouse button may 
define the first comer (for example, the upper-left 
comer) and releasing the left mouse button may 
define the second comer (for example, at the lower 
right) of the bounding rectangle. When the left 
mouse button is initially clicked, the global variable 
selectMode is set to tojbejcreated. In one 
embodiment, the hotspot 8003 is created upon 
release of the mouse button and further movement 
of the pointer. In one embodiment, an EDrawRect 
42,006 object, and corresponding EClipObject 
48,026 and EFrameObject 48,021 objects are 
instantiated in the object database 1 127, described 
in Figure 48, when a hotspot 8003 is created. In 
one embodiment, selectMode may be set to size 
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10 



EPolyTool 43,005 



15 



20 



25 



during hotspot 8003 definition. In one 
embodiment, pointer movement after hotspot 
creation may be handled as a drag of the handle 
wherein ESelectTool=s 43,006 method is called as 
described above. In one embodiment, upon release 
of the mouse button, the physical size of the 
rectangle is checked, and if the physical size is 
determined to be too small, the corresponding 
hotspot 8003 objects are deleted. 
In one embodiment, this tool may be used for 
drawing polygon-shaped hotspots 8003. One 
method for defining polygon-shaped hotspots 8003 
will now be described. In one embodiment, clicking 
a mouse button may define a new EDrawPoly 
42,007, which, as described above, creates the 
corresponding EClipObject 48,026 and 
EFrameObject 48,021 objects in the object 
database 1 127. In one embodiment, hotspot 8003 
definition includes clicking the mouse button when 
the pointer is on the first vertex and dragging the 
pointer to a second vertex. In one embodiment, 
additional vertices of the polygon are defined by 
clicking and dragging the pointer location, as 
describe above. Double-clicking the left mouse 
button causes the last vertex to be deleted. The 
hotspot 8003 is deleted if it does not satisfy a 
requirement for minimum vertices. 



ESetTargetTool 43,008 



30 



In one embodiment, this tool may be used 
for linking a target 7003 to a selected 
hotspot 8003. A method for linking a 
target 7003 to a hotspot 8003 will now be 
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described. In one embodiment, the hotspot 
8003 may be selected by the author making 
a first mouse click while the pointer is 
positioned on the hotspot 8003 in the 
5 Workshop Window 8001. In one 

embodiment, movement of the mouse may 
begin the drag and drop operation. The 
pointer may be dragged over to the Media 
Warehouse window 1 0,003 and the pointer 

1 0 positioned over a media element thumbnail 

4007. In one embodiment, the media 
element thumbnail 4007 may be 
automatically selected and by releasing the 
left mouse button. Then, the drag and drop 

15 operation is completed. Also, the media 

element to be linked as a target 7003, to the 
previously selected hotspot 8003, is 
defined. 



20 After using any one of the draw-tools, the current tool reverts back to selection. 

Undo In one embodiment, the undo mechanism may allow the author to undo 
operations performed in the workshop window 8001 . In one 
embodiment, three types of operations may be undone: (1) moving a 
hotspot 8003 in the frame of the video or image, (2) deleting a hotspot 

25 8003 and (3) changing the Z-Order (layering) of a hotspot 8003. The Z- 

Order is a term known to persons skilled in the art and determines how 
hotspots 8003 are placed upon one another when two or more hotspots 
8003 overiap a region in a frame of video or image. In one embodiment, 
the definition of a hotspot 8003 may also be undone. In one embodiment, 

30 hotspot 8003 interpolation may also be undone. 
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In one embodiment, the EClipView object 43,002 may store an array of 
EUndoCommand 43,017 objects, wherein each array stores the data of 
one operation performed in the workshop window 8001 . In one 
embodiment, the undo function is multi-level, meaning that more than one 
5 operation may be undone. In one embodiment, memory considerations 

may make it advisable to limit the number of undoing operations available. 
The undo array may be deleted every time a new media is brought into the 
Workshop Window 8001. In one embodiment, the undo command may 
be performed regardless of the frame in which the operation was 
10 performed. In otherwords, the undo command may be operational for all 

frames, including those frames not currently selected. In one 
embodiment, the performing the undo operation may result in the 
selected frame being the frame in which the undo operation was 
performed. 

15 In one embodiment, an operation performed on a number of selected 

hotspots 8003, is treated as a single operation. Performing the undo 
command to undo that operation affects all previously selected hotspots 
8003. 

In one embodiment, a redo array may be created as part of 
20 EUndoCommand 43,01 7 objects which is stored in the clip view. Upon 

performing a redo command, associated with the redo array, undone 
operations may be reperformed. In one embodiment, the redo array may 
be deleted upon bringing another media into the Workshop window 8001 . 

The following describes the three types of operations that may be undone, 
25 according to one embodiment: 

EUndoObjectMove 43,018 



In one embodiment, this function may store an array of the hotspots 8003 
that were recently moved including the original geometry for each hotspot 
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8003. Upon performing the undo function, restores the original geometry 
of the hotspots 8003. 

EUndoObjectDelete 43,020 

In one embodiment, this function stores an array of the hotspots 8003 that 
5 were recently deleted. Because a hotspot 8003 may be deleted in multiple 

frames of the video, each hotspot 8003 may be associated with an array of 
stored geometries (for a specified range of video frames). Upon calling 
the function UndoQ^ these geometries may be restored. If a hotspot 8003 
is removed (by using the remove option, or by deleting all its 
1 0 EFrameObject objects 48,02 1 ), the hotspot 8003 may be deleted from the 

object database 11 27. In this case, the EClipObject 48,026 object which 
was removed from the object database 1 127, may be stored with the undo 
data, and restored into the object database 1 127 by the UndoQ function. 

EUndoObjectZOrder 43,019 

1 5 In one embodiment, this ftinction may store an array of the hotspots 8003 

for which the Z-Order may have changed. The Z-order specifies the 
layering of hotspots 8003 and is described above. Because the Z-Order 
may only be changed for one hotspot at a time, this array consists of a 
single item. Since layering of hotspots 8003 across multiple video frames 

20 may be available, each hotspot 8003 may be associated with an array of 

stored indices corresponding to each video frame. When the UndoQ 
function is called the hotspot is restored to its original index in all the 
frames. 

Preview Window 

25 This window, according to one embodiment, performs a preview of a 
hypervideo 10,007 by actuating the run-time module 1101 and playing the 
hypervideo 10,007 in a window. In one embodiment, within the Preview 
window, are the document, view, and frame classes, described below: 
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view 
frame 

5 

Pialpgs 

In one embodiment, operations may be performed with dialogs and 
property sheets. Dialogs and property sheets are known to persons skilled in the 
art of Windows (TM) programming. Each property sheet may be implemented 
10 by a class. 

In the object oriented representation of the authoring tool GUI, the class 
associated with a property sheet is not aware of the classes associated with the 
pages of the property sheet. In one embodiment, the settings property sheet is 

1 5 aware of the classes associated with the pages of the property sheet. The pages 
are added to the property sheet by the class that created it (and not by the class of 
the property sheet). This technique may allow a particular property sheet to be 
used for a some multiple uses, wherein each use adds different property pages 
having different attributes. For example, in one embodiment, a single media 

20 property sheet class may be used for all media types and yet each media type may 
have different pages. 

In one embodiment, the authoring tool 1001 may utilize the following dialogs and 
property sheets: 

25 Media property gheet 

The media property sheet 5003 illustrated in Figure 5 may be used to view 
and change information about a media element. The media property sheet 
5003 may be created and managed by an EMediaProperties object. 

In one embodiment, the media property sheet may include up to five 
30 pages wherein each property page is associated with a class. The pages in 
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the document may be described by class EClipDoc 42,003, like the 
workshop window 

the view may be described by class EClipPreView 
the frame may be described by class EclipPreviewFrame 
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the property sheet may be a ftinction of the type of media the pages 
represent. For example, the five media property page classes may include: 

1 . EMPGeneralPage Property Page ( illustrated in Figure 5) 

This is the general page, which in one embodiment, exists for all 
5 media types (except may be for URL address media). In one 

embodiment, this page may contain general information about the 
media, including, at least, type 5019, subtype, and compression 
5017. In one embodiment, the author may change the name of the 
media element with this page. Because the dialog remains the 
1 0 same for all media types, and yet the information displayed for 

each media type is different, the dialog may contain fields having 
general names including header 1, content 1, header!, content^ etc.. 
When the dialog is opened, these fields may be filled with the 
appropriate data, for example, according to the media type. 

15 2. EMPPreviewPage Property Page 

In one embodiment, this page may enable the author to preview 
the media, including bitmap, text, video and sound media types. In 
one embodiment, the media may be displayed using a media- 
window 6001 (except for bitmap media, which may be displayed 
20 on the page). 

3 . EMPLinksPage Property Page 701 1 

In one embodiment, this page displays a list-control (i.e. MFC 
control known to persons skilled in the art) containing all the 
hotspots 8003, which are defined in the media. In one 
25 embodiment, using a media-display method, the hotspots 8003 are 

displayed as a thumbnail of the media 4007 in the first frame in 
which the hotspot is defined. Double clicking the mouse button 
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on a hotspot 8003 may open the property sheet associated with 
the hotspot 8003. 

4. EMPDefaultsPage Property Page 

In one embodiment, this page provides default cursors data for the 
5 video and bitmap. 

5 . EMPPublishPage Property Page 

In one embodiment, this page allows an author to determine how 
the hypervideo media files may be published and retrieved for use 
in a hypervideo 1 0,007. This page may be applicable for all media 
1 0 types. 

URL media which were created by the author (using the Create new media menu 
option) have only a General page (e.g. EMPURLGenPage object). 

Hotspot properties sheet 

A hotspot properties sheet 7101 may be created and managed by an 
1 5 EObjectProperties object. In one embodiment, the hotspot properties sheet 
7101 may include three pages as follows: 

1 . EOPGeneralPage This is the general page. 



20 2. EOPLinksPage 



25 



This page is the target page and, in one 
embodiment, may contain a tab-control of 
target levels and a list-control of the targets 
for the current level. Both tab-control and 
list-control are Windows (TM) API 
controls that are known to persons skilled 
in the art of Windows (TM) programming. 
In one embodiment, drag and drop targets 
7003 and drag and drop levels 7601 is 
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3. EOPVisualPage Property 
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implemented using the below described 
Drag & Drop methods. In one 
embodiment, when the add-new-target icon 
is dragged onto a media element thumbnail 
4007 in the media window 10,003, the 
media window 10,003 may open the Target 
properties sheet described in Figure 7e, and 
may automatically add the new target. The 
levels tab 7601 is an EdragableTab class 
object, which is a tab-control which may 
handle dragging and dropping of tabs. In 
one embodiment, changes made to the 
target properties sheet 7001 may be applied 
when the OK button of the target properties 
sheet 7001 is actuated. Addition, deletion 
of and reordering targets 7003 or levels 
7601 may be applied when the OK button 
of the hotspot property sheet 5003 is 
actuated. In one embodiment, this 
functionality is implemented by duplicating 
a copy of the EClipObject 48,026 object 
from the object database 11 27 and changing 
the target for this copy. When the OK 
button is actuated the target levels 48,015 
may be copied to the original object in the 
object database 1 127. 

;e This is the marking 1 1 55 page. 



30 Target prope rties sheet 



-WO 99/10822 



PCT/US98/17444 



168 

In one embodiment, the target properties sheet is handled by an 
ETargetProp object and may contain up to 3 pages: General, Frames/Bounds and 
Display. 

5 In one embodiment, different media types have different property page classes, 
according to the type of the linked media. In one embodiment, the names of the 
classes follows the form: 

ESTxyyyPage wherein x denotes the type of the target media, such as V 
1 0 for video, B for bitmap, etc., and wherein yyy denotes the page (Farms for 
General page, Flace for Display page and Mark for Frames/Bounds pages). 
Using this naming convention, for example, ESTTPlacePage denotes Display 
page for Text target media properties. The General, Frames/Bounds and Display 
pages will now be described: 

15 

2. General page 7801 : This page may handle general information about the link 
(target), and it applies to all types of media. This page is illustrated in Figure 
7e. 

20 2 . Frames page (for video 1 0 1 1 ) and Bounds page (for sound 1015) 790 1 : This 
page may be a media window with a media bar and enable the author to set 
the frame range of the target. This page is illustrated in Figure 7f 

3. Display page (for video, bitmap and text media) 7460: This page is illustrated 
25 in Figure 7g. In one embodiment, a thumbnail of target media is displayed 

over a thumbnail of the active media (in the first frame in which the hotspot 
8003 is defined in the video), and the author may set its position and size. In 
one embodiment, for video target media, the media is drawn in its IN frame 
7903 (as set in the Frames page). In one embodiment, to handle the resizing 
30 and relocating of the linked media, an EResizeObj object is used. In one 

embodiment, if the location is relative to the hotspot, the hotspot' s geometry 
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is drawn on the active media and the linked media thumbnail 1013 is located 
relative to it. In another embodiment, if the location is relative to the mouse 
position, the hotspot's click-cursor 7113 (as set in the General page) is drawn 
in the center of the hotspot 8003, and the linked media location is relative to 
5 this point (in runtime the location will be relative to the actual point where the 
click was performed, not to the center of the hotspot 8003). 

Project Settings sheet 

In one embodiment, this property sheet may handle settings that are a part 
of the hypervideo 10,007 (e.g. an Eproject object 48,001) and are saved 
as part of the project file 1670. The settings of the project setting 
property sheet will be used when the project is run in the run-time module 
1101. 

In one embodiment, each project may have a base-object, which is the 
first EClipObject 48,026 object in the hypervideo 10,007. The targets 
48,009 associated with an EClipObject object 48,026 may be activated 
when the project starts running. In one embodiment, the project settings 
property sheet is implemented as a hotspot property sheet 7101 of this 
base object, and an EObjectProperties object is used. In one 
embodiment, the targets page is the same as the hotspot properties 
Targets page 7001, since, it too, consists of defining targets to the base- 
object. In one embodiment, the Settings and Media Location pages of the 
project settings correspond to the General page 7801 and Marking page 
1 1 55 of the Target properties sheet 7703. 

Settings Properties gheet 

The Settings Properties sheet may handle settings that are not a part of 
the hypervideo 10,007, but rather, a part of the authoring tool 1001 
30 environment. These settings do not affect the operation of the hypervideo 

1 0,007 when it is run in other environments. 
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In one embodiment, default hard*coded settings are set by EObviousDoc 
41,003 when the authoring tool 1001 program is first run on a particular 
computer. In one embodiment, the author may change these settings in 
this property sheet. The author may choose to accept the settings in the 
5 Settings properties sheet by clicking the OK button. In one embodiment, 

the settings are operative for the current instance of the authoring tool 
1001 program. In one embodiment, the next time the program operates, 
the settings are reset. By selecting the Set As Default button, the current 
settings in the settings property sheet are put in the Windows Registry, 
10 known to persons skilled in the art, and these settings will be the default. 

In one embodiment, clicking the Reset button sets the settings in the 
property sheet to the default ones (i.e. taken from the Windows Registry, 
if there are any, or the hard-coded ones, otherwise). This will have no 
effect unless you click the OK button afterwards. 

15 

The Setting properties sheet is handled by an ESettings object. In one 
embodiment, this class provides the buttons Set As Default and Reset at the 
bottom of the property sheet, dispensing with the need for the traditional Apply 
button which is a standard Windows Property Sheet button known to persons 
20 skilled in the art. 

The property pages are handled by the following classes: 

Tracking page ESettingsTrackPage 
25 Advanced page ESettingsAdvancedPage 
Workshop page ESettings WorkshopPage 

Preview page ESettingsPreviewPage 
Magic Wand page ESettingsMagicPage 



30 



Project View 

In one embodiment, the project view 1007 may be implemented as a modal 
dialog. A modal dialog is a type of dialog window that is known to persons 
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skilled in the art. In one embodiment, operations performed by the author in the 

authoring 

tool 1101 with the project view 1007 may change the contents of the other 
windows, for example, the Workshop window 8001 and the Media Warehouse 
5 window 10,003. 

The class which implements the project view 1007 dialog may be 
EProjView. In one embodiment, The project view 1007 dialog may include 
three buttons (Large icons, Small icons, and Text mode) and a tree-control. A 
10 tree control is a Windows API control that is known to persons skilled in the art. 
In one embodiment, the three buttons are.part of a toolbar. This method of 
displaying elements in a tree-control in one of different modes (e.g. Large icons. 
Small icons and Text mode) is frequently used by Windows programmers and is 
known to persons skilled in the art, 

15 

In one embodiment, each node, or icon 1 1 ,003 in the tree control may be 
either a target 7003 or a hotspot 8003. In one embodiment, nodes 1 1 ,003 
located in odd levels of the tree are Targets 7003 and nodes 1 1,003 located in 
even levels are hotspots 8003. The thumbnail icons 1 1,003 of the nodes are 
20 created using the MediaDisplay DLL, that is, hotspot 8003 thumbnails 1 1,003 
with the hotspot 8003 drawn on the nodes. In one embodiment, target 7003 
thumbnails are depicted in a reduced size and may include a frame (according to 
the type of the media). In small icons mode, each type of media may have a 
different icon 1 1 ,003, and all hotspots 8003 may have the same icon 1 1,003. 

25 

A tree may be characterized as infinite where a media links to itself, or to 
another media that ultimately links to itself To avoid the problems associated 
with infinitive trees, in one embodiment, the following rule is used: 

30 if the hotshot 8003 node (A) 1 1 ,003 DOES NOT have an uncle (B) with 

the same hotshot, then descendants of a hotshot node (A) are displayed. 
An uncle is defined as a node (B) 1 1,003 that has an ancestor of this node 
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(A) or a brother of one of its ancestors. In such case, the hotspol node 
(A) may have no children in the tree. The father-child terminology for 
describing nodes relationships in a tree data structure is known to persons 
skilled in the art. 

5 

This rule guarantees that if the children of a certain hotspot node 1 1,003 is 
discarded, then another node with this hotspot is currently visible, although not 

necessarily on the screen, because its father is expanded. 

Locate Media 

1 0 In one embodiment, when a project is opened, and if either the media or 

cursor file can not be found, then the Locate Media dialog window is opened to 
allow the author to help locate the missing file. This dialog is of class 
ELocateMedia. In one embodiment, this is a derived class of CFileDialog, which 
may handle the correct extensions for the files, and allows some additional 

1 5 options. In one embodiment, the main option is the Skip All option (Ctrl+Skip 
button). 

The Application 

In addition to the document templates and the dialogs, the MFC 

20 application mechanism maintains general information concerning the authoring 
tool 1001 application. This is an EObviousApp object (which is a CWinApp 
derived class. The CWinApp class is a standard MFC known to persons skilled in 
the art of Windows Programming). In one embodiment, another general object 
for the application is the main frame of the application, which is an EAppFrame 

25 object (CMDIFrameWnd derived class. The CMDIFrameWnd class is a standard 
MFC class known to persons skilled in the art of Windows Programming). In one 
embodiment, these objects can be accessed at any time by every class in the 
application using the global functions AficGetApp (to get a pointer to the 
application) and AficGetMainWnd (to get the a pointer to the main frame). 
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Hence, general data for the application, which is located in those objects, is 
available globally. 

EQbviousApp 

5 This class may handle the initialization of the application in the function 

Initlnstance. In one embodiment, this class performs numerous functions, 
including creating and displaying the document-templates, reading the registered 
data from the Windows Registry, and opening the splash screen (using the 
CSplashWnd class). A Splash Screen is a term known to persons skilled in the art 

10 and is the first image that pops up when a user starts a software program in 
Windows. This screen frequently includes the name of the application, the 
version, serial number and additional title related information. 

g^ppFrame 

1 5 In one embodiment, the main frame may handle the window commands 

(including opening and closing of windows), and the closing of the application. 

General Features 
Toolbars 

20 In one embodiment, there are three toolbars in the authoring tool 1 001 

which are non-standard MFC toolbars, each of which is an EToolBox 44,003 
derived classes. The three are the Workshop Tools, the Hotspot Tools 9001 and 
the Workshop Bar which is illustrated in Figure 8b and may be attached to the 
workshop window 8001 . 

25 In one embodiment, EToolBox 44,003 is a CDialogBar derived class (The 

CDialogBar class is a standard MFC class known to persons skilled in the art of 
Windows Programming) . In one embodiment, the EToolBox object 44,003 may 
handle an array of buttons, of type EToggleButton 44,005, and a background 
image. In one embodiment, the EToolBox 44,003 object may locate the buttons, 

30 and may handle the painting of the buttons and the background. 
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In one embodiment, EToggleButton 44,005 may be a CButton derived 
class (The CButton class is a standard MFC class known to persons skilled in the 
art of Windows Programming), which may have three states: normal, disabled 
and checked. In one embodiment, each state corresponds to a different 
5 background image which may be displayed when the state is operative. In one 
embodiment, EToggleButton 44,005 also may handle mouse moving and 
actuating. 

1 . The Workshop Tools bar may be of type EToolBox 44,003. 

10 

2. The Hotspot Tools 9001 bar may be of type EObjectsBar 44,006 (derived 
from EToolBox 44,003). In one embodiment, this class adds some 
additional methods to handle the hotspots 8003 combo-box (a combo-box 
is a Windows (TM) API standard control that is known to persons skilled 

1 5 in the art of Windows (TM) Programming). 

3. The Workshop Bar illustrated in figure 8b, is of type EWorkshopBar 
44,008 (derived from EToolBox 44,003). In one embodiment, this class 
adds some additional methods to handle the workshop slider 6007. In one 

20 embodiment, it also uses ESelectionButton 44,009 (derived from 

CButton) for the "Go to-ln" and "Go to-Out" buttons which, In one 
embodiment, instruct the Workshop Window 8001 to display the IN and 
OUT frames of the currently displayed video. In one embodiment, the 
EWorkshopBar object uses the ESlider 44,007 class for the frames slider 

25 6007. ESlider is a CSliderCtri derived class, which, in one embodiment, 

adds some multi-frame selection functionality (The CSliderCtri class is a 
standard MFC class known to persons skilled in the art of Windows (TM) 
Programming). 

30 

Run-Time Mpdule 
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The run-time 1101 modules are responsible for presenting the content of 

an OBV file. In one embodiment, the run-lime 1101 environment consists of the 
following seven modules: 

5 ► Mciva.dll which is the generic run time library 1113 

► V-Active.ocx 1 105 which is an ActiveX control to, for example, embed a 
hypervideo 

10,007 player in an HTML file in a browser. 

► Npvadll 
10 ► Npva.class 

► va32.x32 1 103 which is the Macromedia Director (Macromedia, Inc., San 
Francisco, CA) plug-in. 

^ VActiveRenderer.ax which is a standard DirectShow (Microsoft Corp., 
Redmond, WA) graphics/video renderer. 
1 5 ► VAVisual.ax which is a standard DirectShow (Microsoft Corp., Redmond, 
WA) graph filter. 

MrTVA DT J. TNTRODUCTION 

In one embodiment, the mciva.dll 1 1 13 is the core of all run-time 1101 
20 environments and applications 1 102. It may control the flow of the hypervideo 
10,007 project 48001. In one embodiment, the mciva.dll 1113 maintains the 
communication with the viewer and implements the multimedia operations and 
performs all of the media manipulations 1 125 (e.g. marking 1 155). In one 
embodiment, the mciva.dll 1 1 13 is an applications extension which may have only 
25 one exportable function called DriverProc. One embodiment of a DriverProc 
function will now be described. 

In one embodiment, the prototype of the DriverProc function may be: 
extern "C" LONG FAR PASCAL DriverProc(DWORD dwDriverlD, 
HDRVR hDriver, WORD wMessage, LONG IParaml, 
30 LONG lParam2). 

In one embodiment, the first tv/o parameters are ignored. The parameter 
wMessage determines the operation to be performed by the DriverProc fiinction. 
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In one embodiment, The mciva.dll 1113 may support driver operations and MCI 
1109 operations. 

In one embodiment, the supported driver operations are: load, free, open 
and close which will now be described: 
5 Load performs any one-time driver initialization (wMessage gets 

DRV_LOAD). 

Free performs any one-time shutdown tasks (wMessage gets 
DRV_FREE). 

Open performs initialization, done once each time the driver is opened 
1 0 (wMessage gets DRV_OPEN). 

Close performs any cleanup necessary each time the device is closed 
(wMessage gets DRV_CLOSE). 

In one embodiment, the MCI 1 1 09 operations supported are: Open, 
15 close, play, stop, pause, window, status, signal object, internet, put, enable, 
disable, media, visualize, sample and set flag. These operations will now be 
described: 

MCI OPEN DRIVER: wMessage - MC1_0PEN_DRIVER 
20 IParaml - flags passed with the MCI_OPEN 

command 

lParam2 - a pointer to a 

MCI_DGV_OPEN_PARMS 

In one embodiment, this operation may be called 
25 whenever a new project 48,001 is to be loaded. 

The steps for opening a new project 48,001 may 
include: initialization of global variables, right 
button menu construction (play/stop), preview- 
window construction, hypervideo 10,007 project 
30 file initialization and project 48,001 construction. 

In one embodiment, when constructing the project 
48,001, the hypervideo 10,007 project file version 
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may be checked. In one embodiment, old versions 
may be supported, but presenting an hypervideo 
10,007 project of a newer version then that of the 
object database 1 127 may not be supported. In 

5 one embodiment, after constructing the project 

48,001 , the first level 48,015 of the project 48,001 
may opened and the run-time 1101 level tree 
illustrated in figure 1 3 may be created wherein the 
root 13,006 may be the first target level 48,015 of 

1 0 the project 48,001 . In another embodiment, if 

there is no first level or the opening of the tree fails 
MCI close driver is activated and the operation 
fails. 



1 5 MCI CLOSE DRIVER: wMessage - MCLCLOSE^DRI VER 

IParaml - flags passed with the MCI_CLOSE 
command 

lParam2 - a pointer to a 
MCLDGV_OPEN^PARMS 

20 In one embodiment, if the project 48,001 is not 

already opened, then no operation may is 
performed. In another embodiment, a termination 
procedure may performed. The termination 
procedure may include preview-window 

25 destruction, levels tree 13,004 destruction and 

project 48,001 destruction. 



30 



MCI PLAY: 



wMessage - MCI_^PLAY 

IParaml - flags passed with the MCI^PLAY 

command 
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lParam2 - a pointer to a 

MCI_DGV_PLAY^PARMS 

In one embodiment, if the project 48,001 is not 

already opened, or the driver 1 1 09 is already 

5 playing, then no operation is performed. In one 

embodiment, if the driver 1 109 is in the >pause= 
state, then the levels tree 13,004 may be searched 
and the paused levels 48,01 5 may be played. 
Otherwise, the first level 13,006, (the root of the 

1 0 levels tree 1 3,004) may be played. In one 

embodiment, if MCI NOTIFY is passed in 
IParaml, then a new notification window may be 
created. A notification window is an object known 
to persons skilled in the art of Windows 

1 5 programming. Otherwise, it may use the 

notification window supplied. 

MCI stop: wMessage - MCI_STOP 

IParaml - flags passed with the MCI_STOP 
20 command 

lParam2 B ignored. 

In one embodiment, if the project 48,001 is not 
already opened, or the driver 1 109 is not playing, 
or the driver 1 1 09 is in pause state, then no 

25 operation may be performed. Otherwise, the driver 

1 1 09 may be switched to a stop state. If IParaml 
equals 1 then a message is posted to the 
background window. The actual driver 1 109 stop 
may be activated from the message handler (by 

30 calling the driver=s MCI_STOP command but with 

IParaml equals 0). This mechanism allows 



wo 99/10822 



PCTAJS98/17444 



MCI PAUSE: 
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MCI CLOSE: 
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25 MCI WINDOW: 
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processing of waiting messages before driver 1 109 
is stopped. 

wMessage - MCI_PAUSE 

IParaml - flags passed with the MCI_PAUSE 

command. 

lParam2 B ignored. 

In one embodiment, if the project 48,001 is not 

already opened, or the driver 1 1 09 is not playing, 

then no operation may be performed. Otherwise, 

the driver 1 109 may be switched to a pause state. 

wMessage - MCI^CLOSE 

IParaml - flags passed with the MCI^CLOSE 

command. 

lParam2 B ignored. 

In one embodiment, if a project 48,001 is currently 
opened, it may be closed. Closing a project 48,001 
may include the following steps: Switching the run- 
time tree 1 3,004 to a stop state, closing the run- 
time tree 13,004, destroying the run-time tree 
13,004 object, destroying the project 48,001 
object, destroying the background window and 
sending a message to the notification window. The 
message queue emptying mechanism, as in the 
MCI^STOP command, is implemented. 

wMessage - MCI_WINDOW 

IParaml B the window attribute to be changed. 

lParam2 B a pointer to a 

MCLDGVJA^INDOW__PARMS. 



-wo 99/10822 



PCT/US98/17444 



180 

In one embodiment, this operation may be used to 

set the background window attributes such as: 
Windows visibility and Windows text. 



MCI STATUS: 



wMessage - MCI_STATUS 

IParamlB the driver attribute to be retrieved. 

lParam2 B a pointer to a 

MCLDGV^STATUS_PARMS. 

In one embodiment, this operation may be used to 

retrieve driver attributes. 



10 



MCI PUT: 



15 



MCI INTERNET: 



20 



MCI ENABLE: 



25 



wMessage - MC^PUT 

IParamlB MCI put operation flags. 

lParam2 B a pointer to a 

MCLDGV^PUT^PARMS. 

In one embodiment, this operation may be used for 

repositioning background window. 

wMessage - MCI_INTERNET 
IParamlB The value of the internet flag. 
lParam2 B ignored. 

In one embodiment, this operation may be used for 
setting the value of the internet flag. 

wMessage - MCI^ENABLE 

IParaml B Holds the media object's name. 

lParam2 B Holds the media name. 

In one embodiment, this operation may be used to 

browse the object database 1 127 illustrated in 

figure 48 and enable the appropriate media objects 

48,026. In one embodiment, a hotspot 8003, or 

media object that is enabled may be marked with a 
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MCI MEDIA: 

20 

MCI SAMPLE: 

25 
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marking 1 1 55 filter in the run-time module 1 101 
and may be linked to a target 7003 at run-time 
upon actuating the mouse button (e.g. the mouse 
button is clicked) on the hotspot 8003. 

wMessage - MCI^DISABLE 
IParaml B Holds the object=s name. 
lParam2 B Holds the media name. 
In one embodiment, this operation may be used to 
browse the object database 1 127 illustrated in 
figure 48 and disable the appropriate media objects 
48,026. Disabling a media object, or hotspot 8003, 
may prevent the run-time, for example, from 
marking 1 1 55 the hotspot 8003 at run-time with a 
marking 1 1 55 filter and may prevent the run-time 
module 1 101 to link the hotspot 8003 to a target 
7003 upon actuating a mouse button on the 
hotspot 8003. 

wMessage - MCI MEDIA 

IParaml B Holds the object's name. 

lParam2 B Holds the media name. 

In one embodiment, this operation may return 

Boolean TRUE' if the specified media is currently 

playing. 

wMessage - MCI_SAMPLE 

IParaml B may hold a pointer to the video object. 

lParam2 B may hold a pointer to a 

SAMPLE^DATA. 

In one embodiment, this operation may be called by 
the renderer (described above) or the marking 1 155 
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filter (described above) upon rendering a video 
frame. In another embodiment, this operation may 
be used for time calculations and marking 1 155. 



MCI DESTROY NOTIFY: 

5 wMessage - MCI_DESTROY_NOTIFY 

IParam IB ignored- 
IParam2 B ignored. 

In one embodiment, this operation may be used for 
destroying the notification window. 

wMessage - MCLSET^FLAG 
IParaml B the flag to update. 
lParam2 B the new value of the flag. 
In one embodiment, the following flags may be 
updated using this MCI command: 

1 . *use DirectShow' (when TRUE BDirectShow 
is used, when FALSE B MCI is used) 

2, 'allow visualization' and 'run-time 
environment' (when TRUE B the run-time 
environment is not the program authoring tool 
1001). 



10 MCI SET FLAG: 



15 



THE RIJN-TIMH TREE 13.004 

An exemplary design of the run-time tree will now be described: In one 
embodiment, while previewing an hypervideo 10,007 project 48,001 file, the run- 
25 time module 1 101 may maintain a levels tree 1 3,004. In one embodiment, the 
nodes of the run-time tree 13,004 may be the currently active target levels 48,015 
(i.e. Target Levels 48,01 5 that are currently opened). In another embodiment, 
each node represents a target level 48,015 and may hold an array of objects 
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13,002. Each object 13,002 may control the playing of a media file. In one 
embodiment, the type of the media file may be, for example, one of the following: 

Video 101 1, Audio 1015, Bitmap 1013, Text 1017, Executable (.exe file) 

1021, and URL 1019. 
5 In one embodiment, each operation performed on a level 48,015 (e.g. stop, play, 
open etc.) may also be performed on each object 13,002 in that objects array. In 
one embodiment, the root of the tree may be a dummy node 13,006 that may 
have no objects 13,002 in its objects array. In one embodiment, when a new 
project 48,00 is opened (during the execution of MCI^OPEN command) the first 
1 0 real node 1 3,002 may be constructed from the default target level 48,0 1 5 of the 
project 48,001 (known as the Base Target and is constructed as a son of the 
dummy node). In another embodiment, each level (node) may support the 
following operations: Open, Close, Play, Stop, DriverPlay, DriverStop, Pause, 
Resume. Additional operations may be: 



15 

Window 

13,002 

ParentLevel 

20 TargetLevel 

NextTargetLevel 

MainTargetlndex 
25 Previous 



IsMainTarget 



30 DeleteSubTree 



In one embodiment, this operation may set the node's 
window 

In one embodiment, this operation may return a pointer to 
the node's 13,002 parent 

In one embodiment, this operation may return a pointer to 
the node's 13,002 EtargetLevel 48,015 data member. 
In one embodiment, this operation may return a pointer to 
the node's 13,002 next target level 48,015 

In one embodiment, this operation may return a code 

indicating what to do with the previous active node 1 3,002 

(e.g. keep playing, pause, stop, etc.) 

In one embodiment, this operator may check if an object 

13,002 is the main target 48,009 (i.e. the leader). 

In one embodiment, this operation may delete the sub-tree 

under it 
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PlaySubTree 
StopSubTree 
DriverPlaySubTree 



10 



In one embodiment, this operation may play the sub-tree 
under it 

In one embodiment, this operation may stop the sub-tree 
under it 

In one embodiment, this operation may play the sub-tree 
under it as a result of a driver command 
DriverStopSubTree In one embodiment, this operation may stop the sub-tree 

under it as a result of a driver command 
In one embodiment, this operation may close the sub-tree 
under it 



CloseSubTree 



In one embodiment, when a level 13,002 is currently playing the following 
exemplary scenarios can occur: 

1 . One target 48,009 may have finished normal execution (e.g. a video that 
1 5 finished). In one embodiment, when a media finishes playing, it may send 

a message to the notification window. The message handler may query 
the object 13,002 what to do next. An example of options for operations 
to be executed when an object 13,002 finishes playing will now be 
described. In one embodiment the options may be: 
20 Exit In one embodiment, the playback of the project 48,001 

may be stopped and closed using the 
MC1_CL0SE_DRIVER command. 



Loop In one embodiment, this may require no changes because 

25 the default behavior of an object may be set to loop (repeat 

playback). 



Continue In one embodiment, the active level 48,01 5 may be queried 
for the next target level 48,01 5. In another embodiment, if 
30 a next target level 48,01 5 exists, then a new node 1 3,002 

may be constructed from the new target level 48,015 as a 
"brother" to the current active level 48,015. In one 
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embodiment, the current active level 48,015 may be 
stopped. The new level 48,015 may be opened and played. 
In another embodiment, the previous level 48,015 may be 
closed and removed from the tree 13,004. In one 
embodiment, if the playing of the new level 48,01 5 fails, 
then the parent is resumed. 

Back In one embodiment, the current active level 48,01 5 is 

stopped, closed and removed from the tree 13,014. The 
parent level 48,015 is resumed. 

In one embodiment, one of the defined hotspots 8003 may have been 
actuated, for example, by the user clicking the mouse button on the 
hotspot 8003. In one embodiment, hotspots 8003 can be clicked using 
the left button of the mouse. In one embodiment, a message is sent to the 
messages handler for the window (messages, message handlers and 
windows are known to persons skilled in the art of Windows 
(TM)programming). The handler of the window that received the "left 
button down" message may identify the object 13,002 on which the 
clicked hotspot 8003 is defined. In one embodiment, when clicking on a 
hotspot 8003, the configuration of the cursor changes. The object 1 3,002 
retrieves the new cursor configuration and next target level 48,015 by 
contacting the object database 1 127 for the project 48,001 . In one 
embodiment, if the user clicked on an unmarked area, then the object may 
return a null as the new cursor's handle and the new target level 48,015. 
In this embodiment, the new target level 48,01 5 and the new cursor 
48,027 are ignored and no changes occur. In one embodiment, if a new 
cursor 48,027 exists, then it may be presented. In one embodiment, if a 
target level 48,01 5 exists then its type is checked. If it is an exit target 
level 48,015 then the project 48,001 may be stopped and closed. If it is a 
back target level 48,015, then the previous level's 48,015 node 13,002 
may be closed and deleted from the tree 13,004, and its parent may be 
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resumed. Otherwise, a new node 13,014 may be constructed from the 
new target level 48,015, as a son of the current level's node 13,002. The 
new level 48,015 is opened and played. The new level 48,015 may return 
a code indicating what to do with the previous level 48,015. The previous 
5 level 48,015 can be paused, stopped and closed or continue playing. 

3. The entire project 48,001 was paused using the MCI_PAUSE command. 
In one embodiment, the method DriverPause may be activated on each 
node 13,002 in the run-time tree 13,004. In one embodiment, every active 
1 0 media is paused. Later, when the project 48,001 is resumed, each node 

13,002 in the tree 13,004 is resumed. 



One embodiment of the run-time tree class diagram is depicted in figure 45. This 

diagram shows the existence of classes and their relationships. 

15 Figure 45 represents the run-time tree class diagram in general. The different 

items of diagram 45 will now be described in the following table 

Item# Class Name Attributes Operations 
45,009 EVAObject mj5Target:Etargel 

mjjLevel 

45,001 ENode m_oChildren:CobjLisl void DeleteSubTreeQ 

m_pParent:ENode void TakeOfChild() 

void ConnectChildrenToParentO 
void AddChildQ 
void SetParentO 

20 45,008 EVALevel mjTargetLeveliEtargetLevel 

moObj ects : E V AObj ect A rray 

m_oMousePosition:Cpoint 

m_oRefRect:Crect 

mIOffsetFrames: long 
45,007 EVABitmapObject mj3DiB:Edib 

mjDOParmsiEbitmapOParms 

moBitmapStatusiEbitmapStatus 

mJStartTick:long 
4 5 ,006 E V AExecObject m_pOParms : EexecOParms 

iii_oExecStatus:EexecStatus 

mj3ProcessInfo:PROCESS_INF 

ORMATIOM 
45,005 EVASoundObject mjjWaverCwave 

mj)OPamis:EsoundOParms 

m oSoundStatus:ESoundStatus 
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m_pText:char 
m_nTextLength:int 
m jOParms : EtextOParms 
m^oTextStatus: EtextStatus 
m_lStartTick:long 
mj30Panns:EURL0Panns 
m_oURLStatus:EURLStatus 
in_oNavigate:NAVIGATEREC 
m_bTargetAIIocated:BOOL 
m_pMovie:CmyMovie 
m_pMediaNode:EmediaNode 
m_pOPanns:EvideoOParms 
m_oVideoStatus:EvideoStatus 
m_hWaitHandle:HANDLE 
m__sFi leNamerCstring 
mJMediaPositionilong 
m_I lOrgFormatM ed iaPos: LONG 
LONG 

mjjMarking llSSiEVAMarking 
1155 

5 The arrow line between item 45,001 and item 45,008 denotes that 45,008 is 
derived from 45,001. 

Items 45,002, 45,003, 45,004, 45,005, 45,006 and 45,007 are derived from 
45,009. 

10 THK VIDEO OBJECT 

In one embodiment, the video object, implemented by the 
EVAVideoObject class 45,002 may control the presentation of a video file. In 
one embodiment, the EVAVideoObject may hold a pointer to an object of class 
CVAMovie, which is an abstract class that defines the interface to the different 

1 5 objects that actually performs the video operations. The run-time module 1 001 
may, for example, support two multimedia environments: MCI and DirectShow 
(previously called ActiveMovie). 

MCI is one multimedia environment. The run-time module 1001 
20 communicates with the video type MCI drivers by a single API function 

called mciSendCommand and is known to persons skilled in the art of 
Windows programming. 



45,004 EVATextObject 



45,003 EVAURLObject 



45,002 EVAVideoObject 
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DirectShow is another multimedia enviromnent that may control and 
processes streams of time-stamped multimedia data by using modular 
components, called filters, connected in a configuration, called a filter 
graph. This architecture is known to persons skilled in art of Windows 
5 programming. 

The run-time module 1001 may, for example, support the following formats of 

video files: AVI, MPEG-1 and Quick-Time. 

In one embodiment, the following classes control movies in the DirectShow 
1 0 environment: CVAMovieAvi, CV AMovieMpeg and CVAMovieQt each 
corresponding to a particular video format. 

In one embodiment, the CVAMovie may define the following operations: 

1 5 FileOpen In one embodiment, this operation may open the 

video file 

MovieOpen In one embodiment, this operation may open the 

movie in a given window. When opening a video 

20 file in DirectShow the EVAVideoObject 45,002 

first attempts to load the VActiveRenderer filter 
discussed above. In one embodiment, upon 
success, the EVAVideoObject 45,002 may add the 
renderer filter to the DirectShow filter graph. In 

25 one embodiment, upon failure, the 

EVAVideoObject 45,002 may attempt to load the 
marking 1 155 filter. In another embodiment, after 
adding the renderer, or the marking 1 155 filter, to 
the DirectShow graph, the DirectShow graph may 

30 attempt to render the movie file using the new 

filters added. This is standard operation of the 
DirectShow Software Program. If the DirectShow 
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10 



MoviePlay 
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graph cannot render the movie file neither with the 
Tenderer filter nor with the marking 11 55 filter, 
then it may render the movie file anyway using its 
default filters. In this case marking 1 155 may not 
be performed, and the movie is presented using the 
default supplied DirectShow video renderer (which 
may have inferior performance). 
In one embodiment, this operation may put the 
video at a certain position. 

In one embodiment, this operation may play the 
video entirely or play only a specified segment (i.e. 
range of frames). 



1 5 MovieStop 



In one embodiment, this operation may stops the 
presentation of a movie. 



20 



MoviePause 



In one embodiment, this operation may pause the 
presentation of a movie. This means that the movie 
may be stopped and the number of the last frame 
played before the movie is stopped may be saved. 



25 



MovieResume 



In one embodiment, this operation may start playing the 
movie from the position it was previously paused, 
for example, by the MoviePause operation 
described above. 



MovieClose 

30 



In one embodiment, this operation may close the 
currently opened movie file. This may include 
closing of the driver, and deletion of objects. 
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In one embodiment, this operation may get the 
current playing frame number. 



SetSignal 



In one embodiment, this operation may sets the 
video to signal on frames. This means that the 
video playback mechanism will now signal back to 
the object that a frame is being played. 



10 



FrameToTimeFormat 



In one embodiment, this operation may convert 
frame number to playing time, for example, in a 
specified time unit (e.g. milliseconds). 



15 



TimeFormatToFrame 



In one embodiment, this operation may converts 
playing time to frame number. 



One embodiment of the EVAVideoObject 45,002 class diagram is depicted in 
Figure 46. The different features of Figure 46 and the relationships between them 
will now be described in the following table: 



20 



Item # Class Name 



46,001 CVAMovie 



Attributes 



Operations 

FileOpenO 

MovieOpenO 

MovieSetDestOplionsO 

MoviePIayO 

MovieStopO 

MoviePauseO 

MovieCloseO 

FileCloseO 

MovieSeekO 

MovieResumeO 

MovieGelPositionO 

SetSignalQ 

MovieSignalO 

MovieUpdateO 

MovieWindowQ 

FrameToTimeFormatO 

TimeFormatToFrameO 
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46,009 EVideoObject 



20 



46,008 
46,007 

46,005 
46,002 



CVAMovieMciAvi 
CVAMovieMciMpeg 

CVAMovieMciQtw 

CVAMovieAvi 



46,004 
46,003 



CVAMovieMpeg 
CVAMovieQt 



m_pMovie:CVAMovie 

mj)MediaNode:EmediaNode 

mjOParmsiEvideoOParms 

Fn_oVideoStatus:Evideo 

Status 

m_hWaitHandeI:HANDLE 

m_sFileNanie:CString 

m_wDeviccID:WORD 

m_hWnd:HWND 

m_sFileName:CString 

m_wDevicelD:WORD 

m__hWnd:HWND 

m_sFileName:Cstring 

m_IFrameRate:iong 

m_wDevicelD:WORD 

m^hWnd:HWND 

m_sFileName:Cstring 

m_lFrameRate:long 

m_oMilliSecFrameLength:REFTI 

ME 

mi RoundError: long, 

mJlStartTimeiLONGLONG 

m_bUseVisualFilter:BOOL 

m_bUseVActiveRenderer:BOOL 

m_oMediatype:MEDIA__TYPES 

m_pVActiveRenderer:IFiltenn_h 

Wnd:HWND 

in_sFileName:Cstring 

m_IDuration:LONGLONG 

m_INumFrames:LONGLONG 

MJUnitTimeLength:long 



10 The arrow line between item 46,009 and item 46,001 denotes the relationship 
between these items. The arrow line between item 46,008 and item 46,001 
indicates that 46,008 is a class derived from 46,00 L The arrow line between item 
46,007 and item 46,001 indicates that 46,007 is a class derived from 46,001 . The 
arrow line between item 46,005 and item 46,001 indicates that 46,005 is a class 

1 5 derived from 46,001 . The arrow line between item 46,002 and item 46,001 

indicates that 46,002 is a class derived from 46,001 . The arrow line between item 
46,004 and item 46,002 indicates that 46,004 is a class derived from 46,002. The 
arrow line between item 46,003 and item 46,002 indicates that 46,003 is a class 
derived from 46,002. 



THF SniINn OBJECT 

In one embodiment, the sound object , implemented by the EVASoundObject 
class 45,005 may control the playing of wave files. Wave files are sound file 
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format that is known to persons skilled in the art. The EVASoundObject 45,005 
may further direct a pointer to an object of class CWave, which is an abstract 
class that defines the interface to the different objects that actually perform the 
playing of the sound. Sounds can be played using MCI or DirectShow 
environments as discussed above in the case of video playback. The sound object 
EVASoundObject 45,005 may support the following operations: 



Open 



Close 

Play/Stop 

Pause/Resume 

DriverPlay/DriverStop 
Window 



In one embodiment, this operation may open a 

sound file and create a window to receive the 

message sent when the sound finishes. 

In one embodiment, this operation may close the 

sound file, including, deletion of objects. 

In one embodiment, this operation may play or stop 

playback of a sound file. 

In one embodiment, this operation may pause or 

resume the playing of a sound file. 



In one embodiment, this operation may set the 
window for the sound object. 
MediaName In one embodiment, this operation may return the 

media name of the corresponding sound file. 
GetMediaPosition In one embodiment, this operation may get the current 

position in the wave file (e.g. in time units). 
IsPlaying In one embodiment, this operation may return 

Boolean 'TRUE' if the wave file is currently being 
played. 

Previous In one embodiment, this operation may return a 

code indicating what to do with the sound object 
EVASoundObject 45,005 when another object is 
scheduled for playback after it. 

Notify In one embodiment, this operation may return a 

code indicating what to do when a sound object 
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EVASoundObject 45,002 may have finished 
playing. 

One embodiment of the sound object class diagram is depicted in figure 47. The 
different items of diagram 47 and the relationships between them will now be 
described in the following table: 



10 



15 



Item # Class Name 

47.001 CWave 

47,004 EVASoundObject 

47,003 CWaveVA 

47.002 CWaveAM 



Attributes 



m_pWave:Cwave 
mjOParms-.EsoundOParms 
ni_oSoundStatus:ESoundStatus 
m_^wDeviceID:WORD 

ni_j>WinThread:CwinThread 
nij)Graph:IgraphBuilder 
m^hWndiHWND 
mjjLockiCCritSec 



Operations 

LoadO 
PlayO 
StopO 
CloseO 



The arrow line between item 47,004 and item 47,003 denotes the relationship 
between these items. The arrow line between item 47,003 and item 47,001 
indicates that 47,003 is a class derived from 47,001 . The arrow line between item 
47,002 and item 47,001 indicates that 47,002 is a class derived from 47,001 . 



THE TEXT OBJECT 

20 In one embodiment, the text object, implemented by the class EVATextObject 
45,004 may control the presentation of a text file. In one embodiment, when 
playing a text object 45,004 the content of the text file may be painted, for 
example, on a window that was created when the text object 45,004 was opened. 
In one embodiment, the text may disappear after a predefined amount of time. 

25 The text object 45,004 may support the following operations: 
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Open In one embodiment, this operation may open the text file, 

download it from the internet if needed, create a window and set 
its position. 

Play In one embodiment, this operation may switch the window into a 
'show' state which is a Windows API state for a window. This 
terminology is known to persons skilled in the art. In one 
embodiment, this operation may also draw the text on the 
window's device context (Abbreviated *DC' and known to 
persons skilled in the art. In one embodiment, this operation may 
also activate a standard Windows timer to send a message after a 
predefined amount of time. 

In one embodiment, this operation may 'kills' the timer procedure. 



Stop 
DriverPlay 
DriverStop 
Pause 



Resume 



Close 



Update 



Window 



GetTargets 



Timer 



In one embodiment, this operation may 'kill' the timer 
procedure, calculate and save the time remaining for the 
text to be presented. 

In one embodiment, this operation may set a timer 
according to the time calculated in the last pause 
operation. 

In one embodiment, this operation may destroy the text 

window and delete remaining objects. 

In one embodiment, this operation may draw the text on 

the Window's DC discussed above. 

In one embodiment, this operation may sets the text 

object's 45,004 window. 

In one embodiment, this operation may return a pointer to 
an EtargetLevel object 48,015 according to the position of 
the mouse click and the frame number. 
In one embodiment, this operation may return a code 
indicating what to do when the timer (activated during the 
play operation) sends its message. 



wo 99/10822 



PCT/US98/17444 



195 

THE BITMAP OBJECT 

In one embodiment, the bitmap object, implemented by the class 
EVABitmapObject 45,007 may control the presentation of a bitmap file in the 
5 hypervideo 10,007. In one embodiment, when opening a bitmap object 45,007 a 
window may be created. In one embodiment, the size and position of the window 
may be set according to the desired size and position of the bitmap. In one 
embodiment, when playing a bitmap object 45,007 the bitmap may be painted on 
the new window. In addition, according to one embodiment, a timer is created 

1 0 with a time-out value read from the project's 48,001 object database 1 127. 

When the time-out occurs, a message is sent to the object's 45,007 window. In 
one embodiment, the message handler may process the time-out event in a 
manner similar to the MM_MCINOTIFY (sent after a video or a sound file ends. 
MM_MCINOTIFY is a Windows API message known to persons skilled in the 

1 5 art). In one embodiment, the bitmap object 45,007 may hold a pointer to an 

object of class EDib which is responsible for the actual displaying of the bitmap 
(EDib is a service class for opening and decompressing, for example, .bmp, .gif 
and jpg). In one embodiment, the bitmap object 45,007 may support the 
following operations: 

20 

Open 



25 



Play 

30 



In one embodiment, this operation may creates the 
window, set the size and position according to the 
project's 48,001 object database 1 127. In one 
embodiment, this operation may also determine if the 
bitmap file is a local file or if it requires downloading from 
the Internet. In one embodiment, this operation may also 
download the bitmap file if necessary, create an object of 
class EDib, which opens and loads the bitmap file. 
In one embodiment, this operation may paint the bitmap, 
using the EDib object, on the windows device context 
(DC. Discussed above). In one embodiment, this operation 
my also create a timer and set its time-out value according 
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5 

DriverPlay 
DriverStop 
Pause 

10 

Resume 

15 

Close 
Update 
20 IsPlaying 
GetTargets 

25 

MediaName 
Timer 

30 
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to the project's 48,001 object database 1 127, In one 
embodiment, this operation may also notify the parent 
application that the media may have begun playing. 
In one embodiment, this operation may 'kill' the timer 
procedure and notify the parent application. 

In one embodiment, this operation may perform the same 
operation as the Stop operation. In addition, according to 
one embodiment, this operation may store the time 
remaining for the time-out to occur. 
In one embodiment, this operation may play the bitmap 
object 45,007. In one embodiment, this operation may 
also set the timer according to the time calculated in the 
last pause command. 

In one embodiment, this operation may deletes the EDib 

object and destroy the window. 

In one embodiment, this operation may repaints the 

bitmap. 

In one embodiment, this operation may return Boolean 
'TRUE' if the bitmap object 45,007 is currently playing. 
In one embodiment, this operation may return the 
ETargetLevel 48,015 to be opened when, for example, the 
mouse button is clicked on a hotspot 8003 on the Bitmap 
Object 45,007. 

In one embodiment, this operation may returns the media 
name of the Bitmap Media object 45,007 currently being 
handled.. 

In one embodiment, this operation may return a code 
indicating what to do upon successful notification (for 
example, exit, freeze, keep playing). This function is called 
upon timer notification. 
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GetCursor In one embodiment, this operation may return the handle 

of the cursor to be presented. It may return the move 
cursor or the click cursor. 

OnSize In one embodiment, this operation may handle dimensions 

5 changing in the preview window. 

THE URL OBJECT 



In one embodiment, the run-time module 1 101 may support web navigation by 
1 0 activating Targets 7003 that are URL Targets 1019. In one embodiment, the 

viewer can navigate the web by clicking the mouse on hotspots 8003 on video, on 
bitmap or other media. In one embodiment, each time a URL object 45,003 is 
played, a message is sent to the parent application via the notification window. In 
one embodiment, the parent application may be an Internet Browser. In one 
1 5 embodiment, the responsibility for the actual web navigation may be on the 
parent application. In one embodiment, the URL object 45,003 may pass a 
pointer to a NAVIG ATEREC object. The NAVIGATEREC structure may 
contain the URL, the location in the URL and the HTML frame name in case this 
URL is a multi-frame. The URL object 45,003 may support the following 
20 operations: 



30 



Open 



25 Play 



Stop 

DriverPlay 
DriverStop 



In one embodiment, this operation may create the 
NAVIGATEREC object and initialize its fields as read 
from the project's 45,001 object database 1 127. 
In one embodiment, this operation may send the 
NS^NAVIGATE message to parent application, for 
example, Internet Browser with a pointer to the 
NAVIGATEREC object as a parameter. 
In one embodiment, this operation may update the object's 
45,003 status flags. 
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THR KXECUTABLE OBJECT 

In one embodiment, the run-time module 1 101 may support executable 
files as media targets 7003. Executable files can be activated, for example, by 
5 operations such as a hotspot 8003 being clicked on or a Target Level 48,01 5 

finishing execution. In one embodiment, the executable object 45,006 may use the 
CreateProcess standard Windows API fimction to run the executables. In one 
embodiment, the executable object 45,006 may use a timer to periodically check 
the status of the new process. In one embodiment, periodic checking occurs 
1 0 every 1 00 milliseconds. In another embodiment, a worker thread is used rather 
than a timer. In this embodiment, the thread loops repeatedly and with each 
iteration, the status of the new process is checked. The executable object may 
support the following operations: 



1 5 Open In one embodiment, this operation may create a window for 

receiving the timer notifications. In this embodiment the operation 
may also read the executable attributes from the project 48,001 
object database 1127. 
Play In one embodiment, this operation may create a process in which 
20 to run the executable. This operation may also create a timer to 

notify, for example, each 100 milliseconds. 
Stop In one embodiment, this operation may update the status 

parameters of the object 45,006. 
Close In one embodiment, this operation may 'kilP the timer. This 
25 operation may also check the status of the process, and if still 

active, terminate it. In one embodiment, this operation may also 
destroy the notification window. 
Timer In one embodiment, this operation may handle the timer 

notifications. In this embodiment the operation may also check the 
30 status of the new process. 



HOTSPOT MARKING 
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In one embodiment, the run-time module 1101 may support object marking 
1155 when using DirectShow. In this embodiment the marking 1 1 55 may be 
performed on each video frame just before it is rendered. In one embodiment, the 
video frames are passed to the run-time module 1101 by the renderer or the 

5 marking 1 1 55 transform filter. According to this embodiment, if the recent filters 
are not included in the filter graph, then marking 1 155 cannot be performed. 
Further, according to this embodiment, if a defauh DirectShow renderer is used 
then the transform fiUer is automatically added to the graph. 
In another embodiment, marking 1 1 55 may also be performed using the program 

1 0 video renderer. According to this embodiment, if the V-Active video renderer is 
loaded, then it may not be necessary to load the transform filter. Marking 1 155 
may be performed in the run-time module on video samples from the DirectShow 
filter (either video renderer or transform filter) just before rendering. In one 
embodiment, marking 1 1 55 may be performed on samples, for example, of the 

1 5 following color depth: 1 6-bit, 24-bit and 32-bit. 

In one embodiment, marking 1 1 55 may be performed by an object of type 
EVAMarking. According to this embodiment, the EVAMarking object may hold 
a pointer to the video object 46,009 that created it. The pointer may be used to 

20 query the project's 48,001 object database 1 127 whether a hotspot 8003 exists in 
a specific point on a specific frame, and whether this hotspot 8003 should be 
marked 1 155. The following classes inherit from EVAMarking : 
EVA32BitMarking, EVA24BitMarking, EVA555Marking and EVA565Marking. 
These classes perform the different marking 1 155 effects in the supported frame 

25 types. 

Project File 

Active Object database 1 127 
1 . Introduction 

30 In one embodiment, the object database 1 127 may be a software package which 
maintains all the data of a project 48,001 and enables the authoring tool 1001 and 
run-time 1 101 to fetch, modify and serialize this data. In one embodiment, a 



-WO 99/10822 



PCT/US98/17444 



200 

project 48,001 is actually a tree of Targets 48,009. Each Target 48,009 (link) is 
attached to a Media (for example, video 48,002, picture 48,003, sound 48,004, or 
text) and its Operating Parameters 48,023-48,025 (for example, where and how 
the media should be displayed, and what happens when the media starts to 
5 perform and when it ends). However, according to this embodiment, a particular 
media can be attached to more than one target, so the tree may be infinite. In one 
embodiment, a targets-group may contain the targets 48,009 that should be 
performed and the order and manner of their performance. According to this 
embodiment, moving from one targets-group to another is done using hotspots 

10 8003, 48,030. Some media types may contain hotspots 8003. For example, in a 
picture, the hotspot 8003 may be a geometric shape inside the boundaries of the 
picture. In one embodiment, in a video, the hotspot 8003 may also be a geometric 
shape, but it can change its size and shape throughout the frame of the video. In 
this embodiment, the hotspot 8003 may exist only in a part of the video (i.e. only 

1 5 in some of the video frames). In one embodiment, each hotspot object 48,030 in 
the object database 1 127 points to a certain targets-group. This targets-group is 
performed when the hotspot 8003 is clicked using the mouse buttons. 
In one embodiment, another object in the project 48,001 object database 11 27 is 
a cursor object 48,027 which may be representing a standard Windows-cursor. 

20 According to this embodiment, each hotspot 8003 may have two related cursors - 
a move cursor and a click cursor. The move cursor is the cursor that is set when 
the user 1 149 moves the mouse over the hotspot 8003. The click cursor is set 
when the user 1 149 clicks on the hotspot 8003. In one embodiment, in order to 
avoid data duplication, the project 48,001 stores one array of cursors 48,027 and 

25 all the hotspot objects 48,030 hold pointers to indices of this array. 

An exemplary class structure for the project 48,001 object database 1 127 1 127 
will now be described: 

2 Classes 

The following table describes the classes that are defined by the hypervideo 
30 1 0,007 object database 1 127 and their functionality. Figure number 48 shows the 
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class hierarchy and the relationships between the classes in the object database 
1127. 

Class Name # Functionality in this embodiment 



EProject 48,001 

5 EMedia 48,002- 

48,005 

EBitmapMedia 48,003 
EExecMedia 

ESoundMedia 48,004 



ETextMedia 
10 EURLMedia 



EVideoMedia 48,002 

EMediaOParam 48,023- 

48,026 

EBitmapOparams 48,024 
EExecOParanfis 
15 ESoundOParams 48,023 



Contains all the data of the project Contains, for 
example, file name, paths, cursors array, and 
other general information 
Contains data of a certain media. Contains, for 
example, the name of the media file. The actual 
media is read fix>m a file and in one embodiment, 
is not maintained by the object database 1 127. 
EMedia may have a number of derived classes, 
wherein each derived class may contain an 
mstance of its appropriate operating parameters. 
Examples include: 

Contains data of a bitmap type media, 
including, for example, the following file 
types: BMP, DIB, GIF, JPG 
Contains data of an executable file type 
of media element, including, for 
example, the EXE file types 
Contains data of a soxmd type media, 
including, for example, the following file 
types: WAV, AIF, AU 
Contains data of a text type media, 
including, for example, the following file 
types: TXT, RTF 

Contains data of a URL type media, 
including, for example, the HTM and 
HTML file types as well as a string 
containing a URL address on the World 
Wide Web. 

Contains data of a Video type media, 
including, for example, the following file 
types: MPG, AVI, MOV 

Contains data pertaining to Operating parameters 
of a media object. EMediaOParam may have a 
number of derived classes, wherein each derived 
class may contain an instance of its appropriate 
operating parameters. Examples include: 
Contains data pertaining to operating 
parameters of an EBitmapMedia object. 
Contains data pertaining to operating 
parameters of an EExecMedia object. 
Contains data pertaining to operating 
parameters of an ESoundMedia object. 
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10 



ETextOParams 
EURLOParams 
EVideoOParams 
EClip 

EClipObject 
EClipFrame 
EFrameObject 



EGeometry 



EEIlipse 

EPolygon 

ERect 

ETriangle 

ETarget 



ETargetLevel 



1 5 ECursor 



CPointArray 



Contains data pertaining to operating 
parameters of an ETextMedia object. 
Contains data pertaining to operating 
parameters of an EURLMedia object. 

48.025 Contains data pertaining to operating 
parameters of an EVideoMedia object. 

48,022 Contains data about a clip. A clip is a class that 
resides as a member of a, for example, video or 
bitmap media and contains the hypervideo 
10,007 information for the media objects. 

48.026 Contains static information of a hotspot for a 
certain media file. This class holds information 
that is repeated for all instances of hotspots 8003 
throughout the frames of a video. 

48,006 Represents a frame in a video and contains all 
hypervideo information for that frame in the 
video. This object is a member of in an array of 
its type inside an EClip object. 

48,021 Representation of an instance of a hotspot in one 
frame of the video. This class includes a pointer 
to the static hotspot information (i.e. a pointer to 
an EClipObject object) as well as a geometric 
region for the particular hotspot 8003 in the 
particular frame of the video. 
Representation of a geometric shape which 
describes the region of the hotspot in a particular 
frame or group of frames in the video. Derived 
classes of EGeometry, which contain the actual 
size and location of the shape, include: 

48,020 Represents an elliptical region. 
Represents an polygon region. 

48,0 1 7 Represents an rectangular region. 
Represents an triangular region. 

48,009 Contains data pertaining to a target 7003. This 

class may contain the media to be performed and 
its operating parameters. The default parameters 
are the operating parameters of the media object. 

48,01 5 A target level object which is an array of all the 
targets which are performed simultaneously. A 
targets-group is constructed from a list of target- 
levels. 

48.027 Data of a cursor object including its file name and 
a handle to the windows cursor. 

An array of x,y coordinates for use, for example, 
in defining a polygon region. 
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3 



Project Serialization 



In one embodiment, the hypervideo 10,007 Project 48,001 serialization may be 
done using the EProject object 48,001 LoadQ and StoreQ methods. According 
to this embodiment, each of these two functions may have two versions. The first 

5 set of functions may load (and store) the project from (or into) a memory area. 
The second does the same with a file. In one embodiment, these methods use the 
standard serialization procedure of the EProject class. After the project is stored 
in the memory area, the area is compressed using the function ECmprsO. Before 
loading the project from the area, the area must be decompressed using the 

1 0 function EDcmprsO. 

Project Environment 

In one embodiment, general information about the project serialization is stored 
in EProject 48,001 static members. According to this embodiment, this 
information may include the following static member variables which are 
1 5 described in the following table. For the sake of simplicity, the expression 'media 
files ' may also refer to cursor type files: 



mJfRunTime In this embodiment, this member should be set before 
LoadQ is called. Set to FALSE if the project is 



20 



loaded in an authoring tool 1001 environment (that 
is, the authoring tool 1001 reads the project, or the 
run-time of the authoring tool 1001=s preview 



reads the project). Otherwise, set to TRUE. If this 
member's value is set to FALSE, then the media 



25 



30 



relative paths will be added to the path of the OBV 
file {mjsOBVPath) and the custom paths will be 
ignored. That means that the original paths of the 
media files will be used (these are the paths from 
which the author may have read the media in the 
authoring tool 1001). Otherwise, the relative paths 
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m sOBVPath 



10 



m^sVRLPath 

15 



20 

mjsCDPath 

25 

m cCDLetter 
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will be added to the URL and CD paths, and 
custom paths will not be ignored. Thus the publish 
paths are used. Published paths are paths after the 
project may have been published. 

According to this embodiment, this member should 
be set before LoadQ is called and also before 
StoreO is called. This is the path of the OBV file. 
When the project 48,001 is stored, all media paths 
will be relative to this path. When the project 
48,001 is loaded, the paths will be relative to this 
path unless m bRunTime is TRUE. 

According to this embodiment, this member is 
stored within the OBV file. This is the base path for 
media files that come from the Internet. When the 
project is loaded, the paths of such media files will 
be relative to this path unless mJ>RunTime is 
FALSE. 

According to this embodiment, this member is 
stored within the OBV file. This is the base path for 
local media files. When the project is loaded, the 
paths of such media files will be relative to this path 
unless mJbRunTime is FALSE. 

According to this embodiment, this member should 
be set before LoadQ is called. This should be the 
actual drive letter of the CD-ROM in the current 
computer. If the OBV comes from the Internet, 
then the Drive-letter of the CD path will be 
changed to match. In this embodiment, it may be 
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assumed that the Internet OBV will use media from 
the CD. 



m bDownload 



According to this embodiment, this member may be 
stored within the OBV file. This member specifies 
the default source (local or Internet) of media files 
that do not have a specific instruction for 
determining where to get the file from. 



10 



m w Version 



15 



According to this embodiment, this member is set 
by the EProjcct iLoadQ function to the version 
number of the OBV file currently loaded. When 
the project 48,001 is loaded, the serialization 
functions of all the object database 1 127 classes 
check this static member to know the version of the 
OBV. Based on the version number, these 
functions know which data should be stored in the 
OBV. 



20 



25 



mJbForceOBVPath According to this embodiment, this member is 
stored within the OBV file. If the value of this 
member is Boolean TRUE and the project 48,001 
comes from the Internet, then the URL path will be 
set to the OBV path. Otherwise, the CD path will 
be set to the OBV path. 



m_spCursorLoadProc 



30 



According to this embodiment, this member 
should be set before LoadQ is called. This 
member may contain a pointer to a ftinction 
of a form: BOOL LoadCursorProc 
{CString c&a sFileName, CString 
*a_sRealFileName). This function may be 
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used by the run-time module 1 101 to 
download cursors 48,027 that should come 
from the Internet. If this member is NULL, 
the function is not called. 

5 

The Contents of the OBV file 

In one embodiment, the data of the project 48,001 may be compressed using the 
class ECmprsO. According to this embodiment, this class may read a memory area 
and compress it into another memory area. A compression scheme will now be 

1 0 described. In one embodiment, the memory area is scanned byte by byte and each 
0 byte in the input is translated into an 0 bit in the output and each non-zero byte 
in the input is translated into a 1 bit followed by 8 bits of the actual byte. The end 
of the output is signed by a 1 bit followed by eight 0 bits (this combination cannot 
appear inside the output). By way of example, the input: 0, 3, 0, 65, 0, 0, 0, 255, 

1 5 will be translated in the following way: 

Input in bytes: 0 3 0 65 0 0 0 255 

Output in bits: 0 1 000000 11 0 1 0 1 00 000 1 0 0 0 
111111111 10000000 0 

Output in bytes: 64 212 17 

20 255 128 0 

So the output is: 64, 212, 17, 255, 128, 0. 

According to this embodiment, the characteristic of a typical project object 
48,001 imply that the relative number of zeroes in the input memory area is 
expected to be very high. Therefore, the size of the output is expected to be 
25 smaller than the size of the input. 
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3>0 Conclusion 

The present invention is a system and method for authoring and playing 
hypervideo 10,007. The present invention permits creating hyperlinks in select 
regions of interest in frames of a video, defined by hotspots. It is understood that 

5 the above description is intended to be illustrative, and not restrictive. Many 
embodiments will be apparent to those skilled in the art upon reviewing the above 
description. For example, other implementations of computers, using state 
machines, parallel processors, client-server architectures, or combinations 
thereof, may be utilized. Pointing devices, other than mouses, such as pens, 

1 0 joysticks, trackballs and even human fingers, may be used. Displays for 

televisions and computers, may include, but are not limited to, cathode ray tubes, 
plasma displays, and liquid crystal displays. The televisions may utilize either 
analog or digital modulation. The buttons, for example on a mouse, may be any 
type of switch. Switches may also be implemented with capacitively actuated 

1 5 touch screens. The networks may be computer or broadcast (e.g., television 
broadcast) networks. Hence, the scope of the invention should, therefore, be 
determined with reference to the appended claims, along with the full scope of 
equivalents to which such claims are entitled. 
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// NSCommands.cpp : implementation file 
// 

#include "stdafic.h" 
#include "stdio.h" 
#include "NSCommands.h" 

#ifndef PLAYER 

#ifdef_DEBUG 

#define new DEBUG_NEW 

#undefTHIS_FILE 

static char THIS_FILE[] = _FILE_; 

#endif 

#endif 

#ifdef PLAYER 

#defme ASSERT _ASSERT 

#endif 

^define ALLOC_BUFFER_SPARE 40//EVEV - add factor 



#defme ALLOC_HOTSPOTS 3 
#define INT_DEL1MITER 

#define CHAR_TRUE 'Y' 

#defme CHAR_FALSE 'N' 

#defme CHAR_EMPTY 'E' 

#defme TIME_MULTIPL1ER 100.0 



RECT y; 



llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
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// ENSBuffer 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

ENSBuffer::ENSBuffer( BOOL a_bStore ) 
{ 

m_pBuffer = NULL; 
m_iLen = m_iAlloced = 0; 
m_bStore = abStore; 
mJLoadPtr = 0; 

} 

ENSBuffer: :~ENSBuffer() 
{ 

CiearO; 

} 

void ENSBuffer::Clear() 
{ 

if(m_pBuffer !=NULL) 
free ( m_pBuffer ); 
m_pBuffer = NULL; 
m_iLen = 0; 
m_iAlloced = 0; 

} 

BOOL ENSBuffer::Add(const char *a_pBuffer, int aJLen ) 
{ 

if ( m_iAlloced < m_iLen + a_iLen ) 
{ 

// Reallocate. 

int liAlloced = mJLen + aJLen + ALLOC_BUFFER_SPARE; 
char *l_pNewBuffer = (char*)reanoc( mjsBuffer, 

l_iAlloced*sizeof(char) ); 
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if ( IjjNewBuffer = NULL ) 
return FALSE; 

m_pBuffer = l_pNewBufFer; 
miAlloced = liAlloced; 

} 

memcpy( mjsBuffer + in_iLen, a__pBuffer, a_iLen ); 
m_iLen += a_iLen; 

// TRACE( "(%.*s)\n", aJLen, ajaBuffer); 

return TRUE; 

} 

char *ENSBuffer::GetBuffer( int *aJLen/*=NULL*/ ) 
{ 

if(a_iLen !=NULL) 

*a_iLen = m_iLen; 
return m_pBuffer; 

} 

/♦BOOL ENSBuffer::SetString( char *aj3Slring ) 
{ 

ClearQ; 

return AddString( a_pString ); 



BOOL ENSBuffer::String( ENSString &a_oString ) 
{ 

if(Storing()) 
{ 

if(a_oString.Get()) 
{ 
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int len = ( strlen(a_oString.Get()) ); 
Int( len ); 

return Acid( a_oString.GetO, len ); 

} 

else 
{ 

char c = CHAR_EMPTY; 
return Add( &c, 1); 

} 

} 

else 

{ 

if (miLoadPtr >= miLen ) 
{ 

ASSERT(FALSE); 
return FALSE; 

} 

char acNext = mj3Buffer[m_iLoadPtr]; 
if ( a_cNext = CHAR_EMPTY ) 
{ 

a_oString.Set( NULL );//EVEV - check 
miLoadPtr ++; 

} 

else 

{ 

int len; 
lnt( len ); 

if (m_iLoadPtr+len-l >= m_iLen ) 
{ 

ASSERT(FALSE); 
return FALSE; 



Copyright 1998 Veon, Ltd. 



wo 99/10822 



PCT/US98/I7444 



213 

} 

a_oString.Set( m_pBufFer+in_iLoadPtr, len ); 
miLoadPtr += len; 



} 

return TRUE; 



} 



BOOL ENSBuffer: :Bool( BOOL &a_bBool ) 
{ 

if(Storing()) 
{ 

char val = a_bBool? CHAR_TRUE : CHAR_FALSE; 
return Add( &vai, 1 ); 

} 

else 
{ 

if (mJLoadPtr < m_iLen ) 
{ 

if ( m_pBuffer[m_iLoadPtr] = CHAR_TRUE ) 



a bBool = TRUE; 



else if ( m_pBuffer[m_iLoadPtr] == CHAR_FALSE ) 



a bBool = FALSE; 



else 



ASSERT(FALSE); 
return FALSE; 



m iLoadPtr-H-; 
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return TRUE; 

} 

ASSERT(FALSE); 
return FALSE; 

} 

} 



BOOL ENSBuffer::Int( int &a_ilnt ) 
{ 

if(Storing()) 
{ 

charbufllO]; 

sprintf( buf, "%d%c", ajlnt, INT_DELIMITER ); 
return Add( buf, strlen(buf) ); 

} 

else 

{ 

int l iSign = 1; 
ailnt = 0; 

for ( ;m_iLoadPtr < mJLen && m_pBuffer[m_iLoadPtr] != 
INT_DELIMITER; 

mJLoadPtr ++ ) 

{ 

if ( mj3Buffer[m_iLoadPtr] = '-' && 
a_ilnt == 0 ) 



{ 



} 

else 
{ 



// Minus sign allowed on the beginning. 
l_iSign = -l; 



if ( m_pBuffer[m_iLoadPtr] < '0' || 

m_pBuffer[mJLoadPtr] > '9' ) 

{ 
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ASSERT(FALSE); 
return FALSE; 

} 

ajint = ajint * 10 + m_pBuffer[m_iLoadPtr] - *0'; 

} 

} 

a_ilnt *= IJSign; 

mJLoadPtr ++; // Delimiter 

} 

return TRUE; 



} 



BOOL ENSBuffer::Long( long &aJLong ) 
{ 

int temp = a_lLong; 
BOOL l_bRC = lnt( temp ); 
a_lLong = temp; 

return l_bRC; 

} 



IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIHIIIIIIIIIIIIIIIIIIIIII 
II ENSString 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

ENSString: :ENSStringO 
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:ENSBuffer(TRUE) 



ENSString: :~ENSString() 

{ 
} 

BOOL ENSString: :Set( const char *a_pString ) 
{ 

ClearO; 

if ( a_pString ) 

return Add( a_pString, strlen(aj3String) + 1 ); 

return TRUE; 

} 

BOOL ENSString::Set( const char *aj5String, int a_iLen ) 
{ 

ClearQ; 

if ( a_pString ) 

{ 

Add( ajDString, a_iLen ); 
return Add( "", 1 ); 

} 

return TRUE; 

} 

char *ENSString::Get() 
{ 

return GetBufferQ; 

} 
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llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

II ENSFramelnfoManager 

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

ENSFramelnfoManager::ENSFramelnfoManager() 
{ 

m_pInfo = NULL; 

} 
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ENSFrameInfoManager::~ENSFrameInfoManager() 
{ 

if(m_pInfo!=NULL) 
delete m_pInfo; 

} 

BOOL ENSFrameInfoManager::NewInfo( ENSFramelnfo *a_pNewInfo, 
ENSBuffer &a_oBuffer) 

{ 

// Here we may pass only the difference! 

if(m_pInfo!=NULL) 
delete m_pInfo; 

m_pInfo = a_pNewInfo; 

// Create the command. 
aoBuffer.ClearO; 

// write command version. 

int IJVersion = VANS_CMD_VERSION; 

a_oBuffer.Int( IJVersion ); 

if (!m_pInfo->Serial( a_oBuffer ) ) 
return FALSE; 

// Make it null terminated. 
a_oBuffer.Add("", 1); 

return TRUE; 

} 

BOOL ENSFrameInfoManager::NewCommand( char *a_pCommand ) 
{ 
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ENSBuffer l_oBuffer( FALSE/*Load*/ ); 

I_oBufFer.Add( a_pCommand, strlen( a_pCommand ) ); 

if(m_pInfo !=NULL) 
{ 

delete mjslnfo; 
m_pInfo = NULL; 

} 

// Read command version, 
int l iVersion; 

if ( !l_oBuffer.Int( IJVersion ) || 

IJVersion != VANS_CMD_VERSION ) 

{ 

// Wrong version, 
return FALSE; 

} 

m_pInfo = new ENSFramelnfo; 
return m_pInfo->Serial( loBuffer ); 



void ENSFramelnfoManager::Clear() 
{ 

if(m_pInfo !=NULL) 
{ 

delete m_pInfo; 
m_plnfo = NULL; 

} 
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llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II ENSFramelnfo 
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIHII 



BOOL ENSFrameInfo::Serial( ENSBuffer &a_oBuffer ) 
{ 

// event 

BOOL l_bEvent = ( m_pEvent != NULL ); 
a_oBuffer.Bool( I bEvent ); 
if(l_bEvent) 
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{ 

if ( a_oBuffer.LoadingO ) 

{ 

// We have to allocate 

m_pEvent = new ENSEvent(NULL); 

} 

m_pEvent->Serial( a_oBuffer ); 

} 

// Hotspots count 

a_oBuffer.Int( m JHotspotsCount ); 



// Hotspots 

if ( a_oBuffer.Loading() ) 
{ 

// We have to allocate the hotspots array. 
ASSERT( m_aHotspots == NULL ); 

miHotspotsAlloced = m_iHotspotsCount; 

m_aHotspots = (ENSHotspot**) malloc( 
m_iHotspotsAlloced*sizeof(ENSHotspot*) ); 

if ( m_aHotspots = NULL ) 

return FALSE; 

} 

for (int i = 0; i < mJHotspotsCount; i-H- ) 
{ 

if ( a__oBuffer.Loading() ) 

m aHotspots[i] == new 
ENSHotspot(NULL,NUEL,NULL); 

m_aHotspots[i]->Serial( a_oBuffer ); 

} 

return TRUE;//EVEV - check!! 
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} 

ENSFrameInfo::ENSFrameInfoO 
{ 

m_pEvent = NULL ; 
miHotspotsCount = 0; 
miHotspotsAlloced = 0; 
m_aHotspots = NULL; 

} 

ENSFramelnfo: :~ENSFrameInfoO 
{ 

ClearQ; 

} 

void ENSFrameInfo::ClearO 
{ 

SetEvent(NULL); 

if ( m_aHotspots != NULL ) 
{ 

for ( int i = 0; i < mJHotspotsCount; i++ ) 
{ 

if ( m_aHotspots[i] != NULL ) 
delete m_aHotspots[i]; 

} 

free( m_aHotspots ); 
m_aHotspots = NULL; 

} 

m_iHotspotsCount = 0; 
mJHotspotsAlloced = 0; 

} 
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void ENSFrameInfo::SetEvent( ENSEvent *aj3Event ) 
{ 

if(m_pEvent !=NULL) 
delete m_pEvent; 

m_pEvent = a_pEvent; 



BOOL ENSFrameInfo::AddHolspot( ENSHotspot *a_pHotspot ) 
{ 

if(a_pHotspot==NULL) 
{ 

ASSERT(FALSE); 
return FALSE; 

} 

if ( mJHotspotsAlloced <= mJHotspotsCount ) 
{ 

// Reallocate. 

int 1 iHotspotsAlloced = miHotspotsCount + 1 + 
ALLOC_HOTSPOTS; 

ENSHotspot** l_aNewHotspots = (ENSHotspot**) 

realloc( m_aHotspots, 

l_iHotspotsAlloced*sizeof(ENSHotspot*)); 

if ( l_aNewHotspots == NULL ) 
return FALSE; 

m_aHotspots = laNewHotspots; 
mJHotspotsAlloced = liHotspotsAlloced; 

} 

m_aHotspots[ mJHotspotsCount ] = a_pHotspot; 
m_iHotspotsCount ++; 
return TRUE; 
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iiiiiiiiiiiiiiiiiiiiiiiiiiiii/iiiiiiiiiiiiiiiiiiiiiniiiiiii 

II ENSFrameLinkData 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
ENSFrameLinkData: :ENSFrameLinkData(const char *ajDName) 

{ 

m_sName.Set( a_pName ); 
meType = LINK; 
m_bURL = FALSE; 
m_bASF = FALSE; 
m bSeekTime " FALSE; 
in_bSeekMarker = FALSE; 
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m_dSeekTime = 0.0; 
m_bText = FALSE; 

} 



BOOL ENSFrameLinkData::Serial( ENSBuffer &a_oBuffer ) 
{ 

a_oBuffer.String( m_sNanie ); 

int temp = (int) meType; 
a_oBuffer.lnt ( temp ); 

m_eType = (EType)temp; // If storing -> temp didn't ciianged 



if(m_eType==LINK) 
{ 

a_oBuffer.Bool( m_bURL ); 

if(m_bURL) 

{ 

a_oBuffer.String( m sURL ); 
a_oBuffer.String( m sURLFrame ); 
a_oBuffer.String( m sURLLocation ); 

} 



a_oBuffer.Bool( m_bASF ); 
if(m_bASF) 

a_oBuffer.String( msASF ); 



a_oBuffer.Bool( m_bSeekTime ); 

if ( mbSeekTime ) 

{ 

int time 100; // Store time multiplied by 1 00 



if ( a_oBuffer.Storing() ) 

time 100 = m dSeekTime*TIME_MULTIPLIER; 
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a_oBuffer.Int( time 1 00 ); 



if ( a_oBuffer.Loading() ) 

m dSeekTime = 
((double)timelOO)/TIME_MULTIPLIER; 

} 

aoBuffer.BooK mbSeekMarker ); 
if ( mbSeekMarker ) 

a_oBuffer.String( msSeekMarker ); 



a_oBufFer.Bool( m_bText ); 
if ( m_bText ) 

a_oBuffer.String( m_sText ); 



return TRUE; //EVEV-check!!! 

} 

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II ENSEvent 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

IIIIIIIIIIIIIIUIIIIIIIIIIIIIIIIIIIIIIIIUIIIIIIIIIIIIIIIIII 
II ENSHotspot 
lllllllllllllllllllllllllllllllllllllllllllllllllinilllllll 



ENSHotspot: :ENSHotspot( const char *a_pName, 

const char *a_pDescription, 

ENSGeometry *a_pGeometry ) 

ENSFrameLinkData (a_pName) 
{ 

m_sDescription.Set( a_pDescription ); 
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m_pGeometry = a_pGeometry; 
mlMoveCursorlD = - 1 ; //No cursor 
mJClickCursorlD = - 1 ; //No cursor 

} 

ENSHotspot: :~ENSHotspot() 
{ 

if ( mjjGeometry != NULL ) 
delete m_pGeometry; 

} 

BOOL ENSHotspot: :Serial( ENSBuffer &a_oBuffer ) 
{ 

ENSFrameLinkData::Serial( a_oBuffer ); 
// Description 

a_oBuffer.String( m_sDescription ); 
// Cursors 

a_oBuffer.Long( mJMoveCursorlD ); 
a_oBuffer.Long( mJClickCursorlD ); 

// Geometry type. 

if ( a_oBuffer.Storing() ) 

{ 

// Geometry must exist. 

if ( m j)Geometry == NULL ) 

{ 

ASSERT(FALSE); 
return FALSE; 

} 

// Store type 

int temp = (int)m_pGeometry->Type(); 
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a_oBuffer.Int ( temp ); 

} 

else 
{ 

// Get type 
int temp; 

aoBuffer.Int ( temp ); 

ENSGeometry::EType l_eType = (ENSGeometry::EType)temp; 

// We have to allocate the geometry 

switch (LeType) 

{ 

case ENSGeometry::ELLIPSE: 
POINT pt; pt.x=pt.y=0; 
m_pGeometry = new ENSEllipse(pt,0,0); 
break; 

case ENSGeometry::RECTANGLE: 

RECT rc; rc.left=rc.top=rc.right=rc.bottom=0; 

m jGeometry = new ENSRect(rc); 

break; 

case ENSGeometry::POLYGON: 

m j)Geometry = new ENSPolygon(O); 
break; 

default: 

ASSERT(FALSE); 
return FALSE; 

} 

} 

return m_j)Geometry->Serial( a^oBuffer ); 



} 
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llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II ENSGeometry 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

ENSGeometry: :ENSGeometry( EType a_eType ) 
{ 

m_eType = a_eType; 

} 

ENSGeometry::~ENSGeometry() 

{ 
} 
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// Normalize rect. 

void ENSGeometry::NormalizeRect( RECT &a_oRect) 
{ 

if ( a_oRect.right < a_oRecl.iefl ) 
{ 

long temp = a_oRect.right; 
a_oRect.right = a^oRect.left; 
a_oRect.left = temp; 

} 

if ( a_oRect.bottom < a_oRect.top ) 
{ 

long temp = a_oRect.bottom; 
a_oRect.bottom = a_oRect.top ; 
a_oRect.top = temp; 

} 

} 



iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii 

II ENSEIlipse 

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

ENSEIlipse: :ENSEllipse( POINT a oCenter, long aJVRadius, long aJHRadius) 

: ENSGeometry ( ENSGeometry::ELLIPSE ) 

{ 

m_oCenter = a_oCenter; 
mJVRadius = a_lVRadius; 
mlHRadius = alHRadius; 

} 

BOOL ENSEIlipse: :Serial( ENSBuffer «fea_oBuffer ) 
{ 
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a_oBuffer.Long ( m_oCenter.x ); 
aoBuffer.Long ( m_oCenter.y ); 
aoBuffer.Long ( mlVRadius ); 
a_oBuffer.Long ( ralHRadius ); 



return TRUE;//EVEV-check!! 

} 

BOOL ENSEIlipse::PtInside( POINT& a_oPoint ) 
{ 

// The ellipse is a line or a point, 
if ( mJHRadius == 0 || mJVRadius = 0 ) 
return FALSE; 



// Calculate DeltaX and DeltaY 

float X = (float)a_oPoint.x - (float)m_oCenter.x ; 

float y = (float)a_oPoint.y - (float)m_oCenter.y ; 



// Check if the point is inside the ellipse. 

if((((x/m lHRadius)*(x/m lHRadius)) + 
((y/m_IVRadius)*(y/m_lVRadius))) <= 1.0) 

return TRUE ; 

else 

return FALSE ; 
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IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIHIIIIIIIIIIIIIIIIIIIIIIII 
II ENRect 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

ENSRect::ENSRect( RECT& a_oRect ) 
: ENSGeometry ( RECTANGLE) 

{ 

m_oRect = a_oRect; 
NormalizeRect( moRect ); 

} 

BOOL ENSRect::Serial( ENSBuffer «&.a_oBuffer ) 
{ 

aoBuffer.Long ( moRect.left ); 
a_oBuffer.Long ( m_oRect.top ); 
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aoBuflfenLong ( moRect.right ); 
a_oBuffer.Long ( m_oRect.bottom ); 

return TRUE;//EVEV-check! ! 

} 

BOOL ENSRect::PtInside( POINT& a_oPoint ) 
{ 

return PtInRect( &m_oRect, aoPoint ); 

} 



llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II ENSTriangle 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

/*ENSTrianlge::ENSTrianlge( POINT& a_oCenter, long aJVRadius, long 
aJHRadius) 

: ENSGeometry ( ENSGeometry::ELLIPSE ) 
{ 

moCenter = a_oCenter; 
mJVRadius = aJVRadius; 
mJHRadius = aJHRadius; 

} 

BOOL ENSTrianige::SeriaI( ENSBuffer &a_oBufrer ) 
{ 

aoBuffer.Long ( moCenter.x ); 
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a_oBuffer.Long ( ni_oCenter.y ); 
a_oBuffer.Long ( mJVRadius ); 
aoBuffer.Long ( m_lHRadius ); 

return TRUE;//EVEV-check!! 

} 

BOOL ENSTrianlge::PtInside( POINm a_oPoint ) 
{ 

// The ellipse is a line or a point, 
if ( mJHRadius == 0 1| mJVRadius == 0 ) 
return FALSE; 

// Calculate DeltaX and DeltaY 

float X = (float)a_oPoint.x - (float)m_oCenter.x ; 

float y = (float)a_oPoint.y - (float)m_oCenter.y ; 

// Check if the point is inside the ellipse. 

if((((x/m lHRadius)*(x/m lHRadius)) + 
((y/mJVRadius)'(y/mJVRadius))) <= LO) 

return TRUE ; 

else 

return FALSE ; 

} 

*/ 



Copyright 1998 Veon, Ltd. 



wo 99/10822 



235 



PCT/US98/17444 



llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II ENSPolygon 
IIIIIIIIIIHIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 

ENSPolygon: :ENSPolygon(long aJNumOfPoints ) 
: ENSGeometry ( ENSGeometry::POLYGON ) 

{ 

mJNumOfPoints = aJNumOfPoints; 
if (mJNumOfPoints > 0) 

{ 

m_aPoints = new POINT[m_lNumOfPoinls]; 

} 

else 

ni_aPoints = NULL; 

} 

Copyright 1 998 Veon, Ltd. 



wo 99/1 0822 PCT/US98/1 7444 

236 

ENSPolygon: :~ENSPolygon() 
{ 

if (m_aPoints !=NULL) 
delete m_aPoints; 

} 

BOOL ENSPolygon: :SeriaI( ENSBuffer &a_oBuffer ) 
{ 

a_oBuffer.Long ( mJNumOfPoints ); 

if (a_oBuffer.Loading() ) 
{ 

if (m_aPoints !=NULL) 
delete m_aPoints; 

maPoints = new POrNT[m_lNumOfPoints]; 

} 

for (int i= 0; i< mJNumOfPoints; i++) 
{ 

a_oBuffer.Long( m_aPoints[i].x); 
a_oBuffer.Long( m_aPoints[i].y); 

} 

return TRUE;//EVEV-check!! 

} 

BOOL ENSPolygon: :PtInside( POINT& a_oPoint ) 
{ 

// Implement the algorithm for checking if a point is inside a 

polygon. 



HRGN I hRgn = CreatePolygonRgn ( m aPoints, mJNumOfPoints, 
WINDING); ~ 
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BOOL l_bResult = ( PtInRegion ( l_hRgn, a_oPoint.x, a_oPoint.y )); 
DeleteObject( l_hRgn ); 
return l_bResult; 

}. 

void ENSPolygon: :SetPoinl(long ajindex, POINT& a_oPoint) 
{ 

m_aPoints[a_lIndex] = a_oPoint; 

} 
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// NSCominands.h : header file 

II 

#ifndef_NS_COMMANDS_H 
#define _NS_COMMANDS_H 
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llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

II ENSBuffer 
IIIIIIIIIUIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIHIIIIIIIII 

enum ENSCommandVersion 
{ 

VANS_CMD_VER_UNKNOWN = -1, 
VANS_CMD_VER_BETA1 =0, 

}; 

#define VANS_CMD_VERSION VANS_CMD_VER_BETA1 

class ENSString; 

class ENSBuffer 
{ 

public: 

ENSBuffer( BOOL a_bStore ); 

virtual ~ENSBuffer(); 

void ENSBuffer: :CIear(); 

BOOL Add( const char *a_pBuffer, int aJLen ); 

char *GetBuffer( int *a_iLen=NULL ); 

// BOOL SetStringC char *a_pString ); 

BOOL StringC ENSString &a_oString ); 
// char ♦GetStringO { return (char *)m_pBuffer; }//EVEV-delete 

BOOL Int( int &a_ilnt ); 

BOOL Bool( BOOL «&a_bBool ); 

BOOL Long( long &a_lLong ); 

BOOL StoringO { return m_bStore; } 
BOOL LoadingO { return !m_bStore; } 

private: 
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BOOL in_bStore; 
char *m_pBufFer; 
int m_iLen; 
int m_iAlloced; 
int m_iLoadPtr; 

}; 

class ENSString : private ENSBuffer 
{ 

public: 

ENSStringQ; 
virtual -ENSStringQ; 

BOOL Set( const char *a_pString ); 

BOOL Set( const char *a_pString, int a_iLen ); 

char *GetO; 

}; 



llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II ENSGeometry 
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lllllllllllllllllllllllllllllllllllllllllllllllllllllll^^^^^ 
class ENSGeometry 

{ 

public: 

enum EType 
{ 

UNDEFINED, 
ELLIPSE , 
TRIANGLE , 
RECTANGLE , 
POLYGON 



// Defines an elliptic Geometry 
// Triangular Geometry. 

// Rectangular Geometry. 
// Polygon Geometry. Holds irregular ROIs. 



}; 



ENSGeometry( EType a_eType ); 
virtual -ENSGeometryO; 

virtual BOOL Ptlnside( POINT& a^oPoint ) = 0; 
virtual BOOL Serial( ENSBufFer &a_oBuffer ) = 0; 
static void NormalizeRectC RECT &a_oRect); 



EType TypeQ { return m__eType; } 



private: 

EType m_eType; 

}; 



iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiim 

II ENSEUipse 

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
class ENSEUipse : public ENSGeometry 
{ 

public: 

POINT m_oCenter; 
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long m_lVRadius; 
long m_IHRadius; 

ENSEllipse( POINT a_oCenter, long aJVRadius, long aJHRadius); 
virtual BOOL Ptlnside( POINT& a_oPoint ); 
virtual BOOL Serial( ENSBuffer &a_oBufrer ); 



llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II ENSRect 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
class ENSRect : public ENSGeometry 
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{ 

public: 

RECT m_oRect; 



ENSRect( RECT& a_oRect ); 

virtual BOOL Ptlnside( POINT &a_oPoint ); 

virtual BOOL Serial( ENSBuffer &a_oBuffer ); 



iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiin^ 

II ENSPolygon 
llllllllllllllllllllllllllllllllllllllllllllllllllllllilllll 
class ENSPolygon : public ENSGeometry 

{ 

public: 

long mlNumOfPoints; 
POINT *m_aPoints; 

ENSPolygon( long aJNumOfPoints); 
virtual -ENSPolygonQ; 



virtual BOOL Ptlnside( POINT& a_oPoint ); 
virtual BOOL Serial( ENSBuffer &a_oBuffer ); 
void SetPoint(long ajindex, POINT& a_oPoint); 



////////////////////////////^^^^^^ 

// ENSFrameLinkData 

llllllllllllllllllllllllllllllllllllllllllllllllllllllll^^^^ 

class ENSFrameLinkData 
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{ 

public: 

ENSString m_sName; 

enumEType 
{ 

LINK, 
BACK, 
EXIT, 

}; 

EType m_eType; 

BOOL m_bURL; 
ENSString m_sURL; 
ENSString m_sURLFrame; 
ENSString m_sURLLocation; 

BOOL m_bASF; 
ENSString m_sASF; 

BOOL m_bSeekTime; 
double m_dSeekTime; 

BOOL m_bSeekMarker; 
ENSString m_sSeekMarker; 

BOOL m_bText; 
ENSString m_sText; 

ENSFrameLinkData( const char *a_pName ); 
virtual BOOL Serial( ENSBuffer &a_oBufrer ); 

}; 
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llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II ENSEvent 
llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
#define ENSEvent ENSFrameLinkData 



Copyright 1998 Veon, Ltd. 



-wo 99/10822 



PCT/US98/17444 



246 

uiiiimiiniiniiimiuiiiinimimiiim 

II ENSHotspot 

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIUIIIIIIIIIII 
class ENSHotspot : public ENSFrameLinkData 

{ 

public: 

ENSHotspot( const char *a_pName, const char *a_pDescription, 
ENSGeometry *a_pGeometry ); 

virtual -ENSHotspotO; 

virtual BOOL Serial( ENSBuffer &a_oBuffer ); 

//EVEV do we need this? char *m_pName; 
ENSString msDescription; 
ENSGeometry *m_pGeometry; 
long mlMoveCursorlD; 
long mlClickCursorlD; 

}; 



iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii^^^^ 

II ENSFramelnfo 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiim 

class ENSFramelnfo 
{ 

public: 

ENSFramelnfoO; 
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virtual -ENSFramelnfoQ; 
void ClearQ; 

void SetEvent( ENSEvent *aj3Evenl ); 

BOOL AddHotspot( ENSHotspot *aj)Hotspot ); 

virtual BOOL Serial( ENSBuffer &a_oBuffer ); 

ENSEvent *EventO { return m_pEvent; } 

int HotspotsCountQ { return mJHotspotsCount; } 

ENSHotspot **HotspotsO { return m_aHotspots; } 

private: 

// The version of the command, 
int m_iVersion; 

ENSEvent *m_pEvent; 
int mJHotspotsCount; 
int mJHotspotsAlloced; 
ENSHotspot **m_aHotspots; 



llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

II ENSFramelnfoManager 

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

class ENSFramelnfoManager 
{ 

public: 

ENSFramelnfoManagerQ; 
virtual -ENSFramelnfoManagerO; 

BOOL NewCommand( char *aj)Command ); 
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BOOLNewInfo( ENSFramelnfo *a_pNewInfo, ENSBuffer 
&a_oNSBufrer); 

void ClearQ; 

ENSFramelnfo *Info() { return m_pInfo; } 

private: 

ENSFramelnfo *mj3lnfo; 

}; 



#endif 
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// stdafx.h : include file for standard system include files, 

// or project specific include files that are used fi-equently, but 

// are changed infrequently 

// 

#include <afxwin.h> // MFC core and standard components 
#include <aficexth> // MFC extensions 
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#if !defined(_hvclpfile_h) 
#define hvclpfile_h 
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/* HV Project 

Ephyx Technologies Ltd. 

Copyright © 1994. All Rights Reserved. 

SUBSYSTEM: hvtest.exe Application 
FILE: hvclp.h 
AUTHOR: Avner Peleg 

OVERVIEW 



Class definition for a clip file. 

*/ 

#include <stdio.h> 
#include <string.h> 

#inciude <iostreani.h> 
#include <fstream.h> 

// general macros 

#define HV_OK 0 
#defineHV ERROR -1 



enum HvVideoTypes { HV AVI , HV MPEG , HVJPEG , HV_CINEPAC , 
HV_INDEO , HV_QUICKTIME } ; " 

class HvClipMarker { // a structure to hold a marker for a clip 
public: 

long mark ; // time stamp in frames 
char *id ; // marker's id 
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HvClipMarkerO { id = 0 ; } 

HvClipMarker(long fr , char *name) { mark = fr ; id = ((name = 0 ) ? 0 : 
slrdup(name)) ; } 

HvClipMarker( HvClipMarker& sp) { mark = sp.mark ; id = ((sp.id = 0 ) ? 0 : 
strdup(sp.id)) ; } 

}; 

class HvClipROI { 
public: 

int X ; 

int y ; 

int a ; 

int b ; 

float theta ; 

long id ; // id of the sequence. 

class HvClipROI* next ; 

class HvClipROI* prev ; 

class HvClipROI* nextlnSequence ; 

HvClipROIQ { id = 0 ; nextlnSequence = 0 ; } 

HvClipROKlong handle, int x, int y, int a, int b, float theta, HvClipROI *n, 
HvClipROI *p) 

: x(x) , y(y) , nextlnSequence(O) , 
a(a) , b(b) , theta(theta) , 

next(n), prev(p) , id(handle) {} 
virtual long PtInROI(const int x , const int y ) const ; 

}; 

class HvSequencelterator ; 

class HvSequence { 
friend class HvSequencelterator ; 
public: 

HvSequenceO {name = 0 ; id = 0 ; first = last = 0 ;} 
HvSequence(long ,char*, HvClipROI* , HvClipROI* ) ; 
long id ; 
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char* name ; 
HvClipROI *first ; 
HvClipROI *last ; 

}; 

class HvSequencelterator { 
HvClipROI* current ; 
HvSequence* seq ; 
public: 

HvSequencelteratorO {current =0 ;seq = 0 ;} 
HvSequenceIterator(HvSequence*) ; 
HvClipROI* NextO ; 
HvClipROI* FirstO ; 
HvClipROI* LastO ; 

}; 



class HvClipStruct { 

// debug 

public: 

char *_name ; // identifier of the clip in the project 

char *_mediaFilePath ; // full path of the video data file 

HvVideoTypes _type ; // type of video : AVI,etc.. 

int _numMarkers ; // the number of markers defined 

int numAUocatedMarkers ; // the size of the allocated list 
HvClipMarker* JistOfMarkers ; // the list of markers defined for the clip 

int maxFramesForRoi ; // number of allocated frame slots for ROIS; 
HvClipROI** JistOfRoiByFrame ; // a list of ROIs for each frame of the 

clip 

int allocFrameSlots(const long) ; // allocate new fi-ame slots if needed 
int allocMarkerSlotsQ ; // allocate new marker slots if 

needed 
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int allocSequenceSlotsO ; // allocate new sequence slots if 

needed 



long GetSequenceldO ; // derive an ID for a 

sequence 

int IdExists(const int ) ; // checks whether a sequence with the given id 

exists 

HvSequence* _bagOfSequences ; // list of all sequences defined for the 

clip 

int _numSequences ; // number of 

sequences 

int _numAllocatedSequences ; // number of allocated sequence slots 

public: 

HvClipStructQ : Jype(HV_AVI) {} // empty ctor 

HvClipStruct(int &rc, const char *name, HvVideoTypes type) ; // full 
ctor for definition 

HvClipStruct(int& rc, const char *filePath) ; // full ctor from file 

virtual -HvClipStructQ ; // dtor 

void Write(ostream& os) ; // a write method 

int Write(const char *filePath) ; // a write method into a file 

void Dump(ostream& os) ; // a dump method 

int Dump(const char *filePath) ; // a dump method into a file 

// get & set the clip name 
char *name() { return _name ; } 
void setName{const char *name) ; 

// get & set the clip video file path 

char *mediaFile() { return __mediaFilePath ; } 

void setMediaFile(const char *name) ; 

// add a sequence 
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int AddClipSequence(const char* seqName) ; 
// delete a sequence 

int DelSequence(const char* seqName) ; 
int DelSequence(long seqid) ; 

HvClipROI* GetRoiOfFrame(long fr) { if (fr < maxFramesForRoi) return 
JistOfRoiByFrame[fr] ; 

else return 0 ; } 
char* GetSequence(long seqId) ; 

int GetSequences(char ***names) ; 
int GetSequencesExt(char ***names) ; 

void AddToSequence(HvClipR01* ); // adds a ROI to a sequences 
(always at the end) 

// add a ROI stamp 

int addRoi( long t,long handle, int x, int y, int a, int b, float theta = 0.0 ); 

// add a marker on the clip 

int addMarker( long t,char *id ); 

// delete a marker 

int delMarker(char *id) {return 1;} 

// get an array of markers 

int getMarkers(long *ts, char **ids) {return 1;} 

// find the object at (t,x,y) of the clip 

long ObjectAtFramePoint(const long t,const int x, const int y) const ; 

// clear object^s references 

int delObjectRefs(const long handle) {return 1;} 

}; 
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#endif 

#include "stdafx.h" 
#include <string.h> 
#include "hvflstr.h" 

BOOL EGetStr( FILE* f, char* s, int max_s ) 
{ 

WORD 1; 

// get the Hne length 
if( fread(&l, sizeof(WORD), 1, f) '= 1 ) 
return FALSE; 

if( 1<= 0 II 1+1 > max_s ) 
return FALSE; 

// read the line itself 
if(fread(s, l,l,f) !=1) 
return FALSE; 

for(inti = 0;i<l-l;i++,s++) 

*s = *s^255;. 
*s = 'XO'; 

return TRUE; 

} 



BOOL EPutStr( FILE* f, char* s ) 
{ 

WORD 1 = strlen(s); 

// write the line length 

if( fwrite(&l, sizeof(WORD), 1, 0 != 1 ) 
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return FALSE; 

char *p = s; 

for( int i = 0; i < 1; 1++, p++ ) 
*p = *p'^255; 

// write the line itself 
if( fwrite(s, l,l,f)!=I) 
return FALSE; 

return TRUE; 

} 
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#ifhdef_EFLCSTR_H_ 
#define _EFLCSTR_H_ 

#include "stdich" 

#ifhdefTRUE 
#define TRUE 1 
#endif 

#ifndef FALSE 
#define FALSE 0 
#endif 

typedefint BOOL; 

typedef unsigned short WORD; // 1 6 bits long 

BOOL EGetStr( FILE* f, char* s, int max_s ) ; 
BOOL EPutStr( FILE* f, char* s ) ; 
#endif 
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/* HV Project 

Ephyx Technologies Ltd. 

Copyright © 1994. All Rights Reserved. 

SUBSYSTEM: hvtest.exe Application 
FILE: hvobj.cpp 



OVERVIEW 



Class member methods for an object file. 

file structure: 
<object name> 
<num sequences> 

<sequence> each sequence is <clip name> , 

<identifier> 
*/ 

#include "stdafx.h" 
#include "hvobj.h" 
#include "hvflstr.h" 

static char flstr[256]; 

UllllllliniUIIUIIIIII HvObjSequence lllllllllllllllllllllllllllllllllllll 
II 

II definition ctor 

// 

HvObjSequence::HvObjSequence(const char* clip, const char* seq, 
HvObjSequence* n) : 
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next(n) 

{ 

if(clip!=0) 

clipName = strdup(clip) ; 

if(seq !=0) 

clipSequence = strdup(seq) ; 

} 

///////////////////////////// HvObjectStruct llllllllllllllllllllllllllllllll 
II 

II definition ctor 

// 

HvObjectStruct: :HvObjectStrucl(const char *id) 
{ 

numSequences = 0 ; 
theSequencesListHeader = 0 ; 
theTarget = 0 ; 

// copy the name 
if (id !-0) 

{ 

this->id = strdup(id); 

} 

else 

id = "no^name" ; 

} 

// 

// a ctor from a file path: 

// opens the file and reads the info 

// rc = -1 if fails 

HvObjectStruct: :HvObjectStruct( int &rc, const char *filePath ) 
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{ 

FILE *fileDesc ; 
char line[256] ; 
char line2[256] ; 



rc = HV_OK ; 

if (filePath != 0 && filePath[0] != '\0') 
{ 

// open the file for read only 
if ((fileDesc = fopen(filePath, "rb")) = 0 ) 
{ 

rc = HV_ERROR ; 
return ; 

} 

// get the name of the Object 
if (!EGetStr(fiIeDesc,line,255)) 

{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

// allocate and copy the data 

if ((id = newchar[strlen(line) + 1]) = 0) 

{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

strcpy(id , line) ; 



// read the sequences set for the Object 
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if (!EGetStr(fileDesc,line,255)) 
{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

FILE* fd = fopen("c:\\temp\\moshe.log","a+") ; 
fprintf(fd;'%sMine) ; 
fclose(fd) ; 

sscanf(l ine , "%d" ,&num Sequences) ; 
theSequencesListHeader = 0 ; 
// loop and read all the sequences 
for ( int i = 0 ; i < numSequences ; i++) 

{ 

if ( !EGetStr(fileDesc,flslr,255)) 
{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

sscanf(flstr, "%s %s", line2, line) ; 
theSequencesListHeader = new 

HvObjSequence(line2,line, 

theSequencesListHeader) ; 

} 

// get the target 
int isTargeted ; 
EGetStr(fileDesc,flstr,255) ; 
sscanf(flstr,"%d",&isTargeted); 
if (isTargeted != 0) 
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{ 

theTarget = new char[isTargeted+4]; 
strcpy(theTarget, flstr); 
while (*theTarget != ' ') theTarget-H- ; 
while (*theTarget == ' ') theTarget-H- ; 
theTarget[strlen(theTarget)] = '\0' ; 

} 

else 

theTarget = 0 ; 
fclose(fileDesc) ; 

} 

else 

{ 

rc = HV_ERROR ; 

} 

return ; 



// 

// a virtual dtor 

// 

HvObj ectStruct: :~HvObjectStruct() 
{ 

HvObjSequence* temp; 

if (id != 0) 
delete id ; 

if (theTarget !=0) 
delete theTarget ; 

if (theSequencesListHeader != 0) 
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while ((temp = theSequencesListHeader->next) != 0) 
{ 

delete theSequencesListHeader ; 
theSequencesListHeader = temp ; 

} 

} 

int HvObjectStruct::AddObjSequence(const char* clip, const char* seq) 
{ 

return AddObjSequence(new HvObjSequence(clip,seq,0)); 

) 

int HvObjectStruct::ExistObjSequence(const char* clip, const char* seq) 
{ 

HvObj Sequence* temp = theSequencesListHeader ; 
while (temp != 0) 
{ 

if ((strcmp(temp->clipName, clip) == 0) && 

(strcmp(temp->clipSequence, seq) === 0)) 
return 1 ; 
temp = temp->next ; 

} 

return 0 ; 

} 

int HvObjectStruct::AddObjSequence(HvObjSequence* seq) 
( 

if(seq==0) 

return HV_ERROR ; 

seq->next = theSequencesListHeader ; 
theSequencesListHeader = seq ; 
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numSequences++ ; 
return HV_OK ; 

} 

int HvObjectStruct::GetSequences(HvObjSequence** theSequences) 
{ 

♦theSequences = theSequencesListHeader ; 
return numSequences ; 

} 

int HvObjectStruct::GetSequences(char*** theSequences) 
{ 

*theSequences = new char* [numSequences] ; 

HvObj Sequence* temp = theSequencesListHeader ; 
int total = 0 ; 

while (temp != 0) 
{ 

(* theSequences) [total] = new 
char[strlen(temp->clipName)+strlen(temp->clipSequence)+3] ; 

sprintf((*theSequences)[total+H-],"%s::%s",temp->clipName,temp->clipSequence 

) 9 

temp = temp->next ; 

} 

return numSequences ; 

} 



void HvObjectStruct::DelSequence(const char* clip, const char* seq) 
{ 

HvObj Sequence* temp = theSequencesListHeader ; 
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HvObj Sequence* prevTemp = 0 ; 

while (temp != 0) 

if ( (strcmp(temp->clipName, clip) = 0) && 

(strcmp(temp->clipSequence, seq) = 0) ) // found it 

{ 

if (prevTemp != 0) 

prevTemp->next = tenip->next ; 

else 

theSequencesListHeader = temp->next ; 
numSequences— ; 
break ; 

} 

else 

{ 

prevTemp = temp ; 
temp = temp->next ; 

} 

} 

// 

// a method to write an object file 
// 

int HvObjectStruct::Write(consl char *FilePath) 
{ 

ofstream outfile ; 

if(FilePath = 0) 
return HV_ERROR ; 

outfile,open(FilePath, ios::out ) ; 

Write(outfile) ; 

outfile.closeO ; 
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return HV_OK ; 

} 
// 

// a debug method to dump to file 

// 

int HvObjectStruct::Dump(const char *FilePath) 
{ 

ofstream outfile ; 

if(FilePath==0) 
return HV_ERROR ; 

outfile.open(FilePath, ios::out ) ; 

Dump(outfile) ; 

return HV_OK; 

} 

// 

// a method to write an object file 

// 

void HvObjectStruct::Write(ostream& os) 
{ 

OS « id « "\n" ; 

OS « numSequences « "\n" ; 

for (HvObi Sequence* temp = theSequencesListHeader ; temp != 0 ; temp= 
temp->next) 

OS « temp->clipName « " " « temp->clipSequence « "\n" ; 

if(theTarget !=0) 

OS « strlen(theTarget) « " " « theTarget « "\n" ; 
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else 

OS « "0\n" ; 

} 

// 

// a debug method to dump all info 

// 

void HvObjectStruct::Dump(ostream& os) 
{ 

OS « " dumping an HvObjectStruct" « "\n" ; 
OS « " object name: " « id « "\n" ; 

OS « " sequences defined: " « numSequences « "\n" ; 

for (HvObiSequence* temp = theSequencesListHeader ; temp != 0 ; temp= 
temp->next; 

OS « "from clip: " « temp->clipName « " called: " « 
temp->clipSequence « "\n" ; 

if(theTarget!=0) 

OS « "target: " « theTarget « "\n" ; 

else 

os«" No target !! \n" ; 
OS « " end of dump" « "\n" ; 

} 



#if !defmed(_hvobjfile_h) 
#defme _hvobjfile_h 
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/* HV Project 

Ephyx Technologies Ltd, 

Copyright © 1994. All Rights Reserved. 

SUBSYSTEM: hvtest.exe Application 
FILE: hvobj.h 



OVERVIEW 



Class definition for a clip file. 

*/ 



#include <stdio.h> 
#include <string.h> 

#include <iostream.h> 
#include <fstream.h> 

#include "hvclp.h" 
// general macros 

class HvTarget ; 

class HvObj Sequence { 
public: 

char *clipName ; 

char *clipSequence ; 

HvObj Sequence* next ; 

HvObjSequence(const char* clip, const char* seq, HvObj Sequence* n) ; 
char *Clip() { return clipName ; } 
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char *SequenceO { return clipSequence ; } 



class HvObjectStruct { 
public: //debug 

char* id ; 

char* theTarget ; 



// target 



// identifier of the object 



// HvObj Sequence* theSequencesList ; 
HvObj Sequence* theSequencesListHeader ; 
int numSequences ; 

public: 

HvObjectStructO {id = 0 ; theTarget = 0 ; } 
// empty ctor 

HvObjectStruct(const char* id) ; // full ctor 

HvObjectStruct(int &rc,const char* FilePath) ; // fiill ctor from file 
virtual -HvObj ectStructQ ; // dtor 

int AddObjSequence(HvObjSequence* seq) ; // adds a sequenc 

the object's definition 

int GetSequences(HvObjSequence** theSequences) ; 

int GetSequences(char*** theSequences) ; 

int AddObjSequence(const char* clip, const char* seq) ; // adds a sequence. 

int ExistObjSequence(const char*clip, const char* seq) ; // checks if this 
sequence exists. 

void DelSequence(const char*clip, const char* seq) ; // deletes an existing 
sequence. 

int Write(const char* FilePath) ; // write to file 

void Write(ostream& os) ; // write to a stream 

int Dump(const char* FilePath) ; // dump to file 

void Dump(ostream& os) ; // dump to a stream 
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void SetTarget(const char* tg) { if (tg != 0) theTarget = strdup(tg) ; } 
char* TargetO { return theTarget ; } 

}; 

#endif 
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#include "stdafx.h" 
#include "iostream.h" 
^include "fstream.h" 

^include "hvprj.h" 
#include "hvflstr.h" 

static char flstr[256] ; 
#define M_FLSTR 256 

int HvProject::GetClips(char ***CIips) 
{ 

// open the project file 

FILE *infile; 

int nc,no ; 

int numCUps = 0 ; 

char Hne[256]; 

infile = fopen(path,"rb"); 

if(infile = 0) 
return 0 ; 

// read the first line: <numClips> <numObjects> 
EGetStr(infile,flstr,M_FLSTR) ; 
sscanf(flstr,"%d %d",&nc,&no); 

// allocate string array 
(*Clips) = new char* [20] ; 
if(*Clips==0) 
return 0 ; 

while (EGetStr(infile,flstr,M_FLSTR)) 
{ 
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sscanfi(flstr,"%d %s",&no,line); 
if(no==HV_PRJ_CLIP) 
{ 

(*Clips)[numClips-H-] = strdup(Iine) 

} 

} 

//fill the rest with 0 
for (int i = nc ; i < 20 ; i++) 
(*Clips)[i] = 0; 

fclose(infile); 

return numClips ; 

} 

int HvProject::GetObjects(char ***Objects) 
{ 

// open the project file 
FILE *infile; 
int nc,no ; 

int numObjects = 0 ; 
char line[256]; 

infile = fopen(path,"rb"); 

if(infile = 0) 
return 0 ; 

// read the first line: <numClips> <numObjects> 
EGetStr(infile,flstr,M_FLSTR) ; 
sscanf(flstr,"%d %d",&nc,&no); 

// allocate string array 
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(•Objects) = new char* [20] ; 
if(*Objects = 0) 
return 0 ; 

while (EGetStr(infile,flstr,M_FLSTR)) 
{ 

sscanf(flstr,"%d %s",&nc,line); 
if (nc == HV_PRJ_OBJECT) 

{ 

(*Objects)[numObjects++] = strdup(line) ; 

} 

} 

// fill the rest with 0 
for (int i = no ; i < 20 ; i++) 
(*Objects)[i] = 0 ; 

fclose(infile); 

return numObjects ; 

} 

// Store the project 

int HvProject::Save(int numClips, char **Clips, int numObjects, char **Objects) 
{ 

ofstream outfile; 

outfile.open(path, ios::out) ; 

int nClips = 0 ; 
int nObjects = 0 ; 

// count actual number of clips 
for (int i = 0 ; i < numClips ; i-H-) 
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if(Clips[i] !=0) 
nClips-(-+ ; 

for (i = 0; i < numObjects ; 
if (Objects[i] != 0) 
nObjects++ ; 



outfile « nClips « " " « nObjects « "\n" ; 

for (i = 0 ; i < numClips ; i++) 
if(Clips[i] !=0) 

outfile « HV_PRJ_CLIP « " " « Clips[i] « "\n" ; 

for (i = 0 ; i < numObjects ; i-H-) 
if(Objects[i] !=0) 

outfile « HV_PRJ_OBJECT « " " « Objects[i] « "\n" ; 

outfile.closeO ; 

return 1 ; 

} 

int HvProject::SaveAs( int numCIips, char **Clips, int numObjects, char 
♦♦Objects, const char *path ) 

{ 

delete this->path ; 
this->path = strdup(path) ; 

return Save(numClips, Clips, numObjects, Objects) ; 

} 



#if !defmed( ^hvprjfile_h) 
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#define ^hvprjfile_h 

/* HV Project 

Ephyx Technologies Ltd. 

Copyright © 1994. All Rights Reserved. 

SUBSYSTEM: hvtestexe Application 
FILE: hvprj.h 



OVERVIEW 

Class definition for a project file. 

*/ 



^define HV_PRJ_CLIP 0 
#define HV_PRJ_OBJECT 1 

#include <string.h> 

class HvProject 
{ 

public: 

char *path ; // the path of the project directory 

public: 

HvProject( char *p ) {path = strdup(p); ) 

// Export the project data: clips, objects etc. 
int GetClips(char ***Clips) ; 
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int GetObjects(char ♦♦♦Objects) ; 
char^ PathQ { return path ; } 
// Store the project 

int Save(int numClips, char ♦♦Clips, int numObjects, char ♦♦Objects) ; 

int SaveAs( int numClips, char ♦♦Clips, int numObjects, char ♦♦Objects, 
const char ♦paUi ) ; 

}; 

#endif 
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// hvstrdoc.cpp : implementation of the CHvstreamDoc class 
// 

#include "stdafx.h" 
#include "hvstream.h" 

#include "hvstrdoc.h" 

#ifdef_DEBUG 
#undefTHIS_FILE 

static char BASED_CODE TH1S_FILE[] = _FILE_; 
#endif 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiniiim 

II CHvstreamDoc 

lMPLEMENT_DYNCREATE(CHvstreamDoc,CDocument) 

BEGIN_MESSAGE_MAP(CHvstreamDoc,CDocument) 
//{ { AFX_MSG_MAP(CHvstreamDoc) 

// NOTE - the ClassWizard will add and remove mapping macros 

here. 

// DO NOT EDIT what you see in these blocks of generated 

code! 

//}}AFX_MSG_MAP 
END_MESSAGE_MAP() 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II CHvstreamDoc construction/destruction 

CHvstreamDoc: :CHvstreamDoc() 
{ 

// TODO: add one-time construction code here 

} 
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CHvstreamDoc: :~CHvstreamDoc() 

{ 
} 

BOOL CHvstreamDoc: :OnNewDocumentO 
{ 

if (!CDocument::OnNewDocumentO) 
return FALSE; 

// TODO: add reiriitialization code here 
// (SDI documents will reuse this document) 

return TRUE; 

} 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiniiiiiiuiiiiiiiiiiiiiiii 

II CHvstreamDoc serialization 

void CHvstreamDoc::Serialize(CArchive& ar) 
{ 

if(ar.IsStoring()) 
{ 

// TODO: add storing code here 

} 

else 
{ 

// TODO: add loading code here 

} 

} 



lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
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// CHvstreamDoc diagnostics 
#ifdef_DEBUG 

void CHvstreamDoc::AssertValid() const 
{ 

CDocument: : AssertValidQ; 

} 

void CHvstreamDoc: :Dump(CDumpContext& dc) const 
{ 

CDocument: :Dump(dc); 

} 

#endif//_DEBUG 



///////////////////////////W^^^^ 

// CHvstreamDoc commands 
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// hvstrdoc.h : interface of the CHvstreamDoc class 
// 

/////////////////^^^^^^ 

class CHvstreamDoc : public CDocument 
{ 

protected: // create from serialization only 
CHvstreamDocO; 

DECLARE_DYNCREATE(CHvstreamDoc) 

// Attributes 
public: 

// Operations 
public: 

// Overrides 

// Class Wizard generated virtual function overrides 

//{ { AFX_,VIRTUAL(CHvstreamDoc) 

public: 

virtual BOOL OnNewDocumentQ; 
//}}AFX_,VIRTUAL 

// Implementation 
public: 

virtual ~CHvstreamDoc(); 

virtual void Serialize(CArchive& ar); // overridden for document i/o 
#ifdef_DEBUG 

virtual void AssertValidQ const; 

virtual void Dump(CDumpContext& dc) const; 

#endif 
protected: 
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// Generated message map functions 
protected: 

//{ { AFX_MSG(CHvstreamDoc) 

// NOTE - the Class Wi2ard will add and remove member functions 

here. 

// DO NOT EDIT what you see in these blocks of generated 

code ! 

//}}AFX_MSG 

DECLARE_MESSAGE_MAP() 

}; 

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIU 
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// hvstream.cpp : Defines the class behaviors for the appHcation. 
// 

^include "stdafx.h" 
^include "hvstream.h" 

#include "mainfrm.h" 
#include "hvstrdoc.h" 
#include "hvstrvw.h" 

#ifdef_DEBUG 
#undefTHIS_FILE 

static char BASED_CODE THIS_FILE[] = _FILE_; 
#endif 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiin 

II CHvstreamApp 

BEGIN_MESSAGE_MAP(CHvstreamApp, CWinApp) 
//{{AFX_MSG_MAP(CHvstreamApp) 
ON_COMMAND(ID_APP_ABOUT, OnAppAbout) 

// NOTE - the Class Wizard will add and remove mapping macros 

here. 

// DO NOT EDIT what you see in these blocks of generated 

code! 

//}}AFX_MSG_MAP 
// Standard file based document commands 
ON_COMMAND(ID_FILE_NEW, CWinApp: rOnFileNew) 
ON_COMM AND(ID_FILE_OPEN, CWinApp: :OnFileOpen) 
END_MESSAGE_MAP() 

lllllllllllllllllllllllllllllllllllllllllllllllllllilllllllllllllllllllllllll 
II CHvstreamApp construction 

CHvstreamApp : : CH vstreamAppQ 
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{ 

// TODO: add construction code here, 

// Place all significant initialization in Initlnstance 

} 

llllllllllllllllllllllllllllllllllllllllllllllim 

II The one and only CHvstreamApp object 

CHvslreamApp theApp; 

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll^ 
II CHvstreamApp initialization 

BOOL CHvstreamApp: :Initlnstance() 
{ 

// Standard initialization 

// If you are not using these features and wish to reduce the size 
// of your final executable, you should remove from the following 
// the specific initialization routines you do not need. 

Enable3dControls(); 

LoadStdProfileSettingsO; // Load standard INI file options (including 

MRU) 

// Register the application's document templates. Document templates 
// serve as the connection between documents, frame windows and views. 

CSingleDocTemplate* pDocTemplate; 

pDocTemplate = new CSingleDocTemplate( 
IDR_MAINFRAME, 
RUNTIME_CLASS(CHvstreamDoc), 
RUNTIME_CLASS(CMainFrame), // main SDl frame 

window 

RUNTIME_CLASS(CHvstreamView)); 
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AddDocTemplate(pDocTemplate); 

// create a new (empty) document 
OnFileNewQ; 

if(mJpCmdLine[0] !=^0') 
{ 

// TODO: add command line processing here 

} 

return TRUE; 

} 

lllll/lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
!l CAboutDlg dialog used for App About 

class CAboutDlg : public CDialog 
{ 

public: 

CAboutDlgO; 

// Dialog Data 

//{ {AFX_DATA(CAboutDlg) 
enum { IDD - IDD_ABOUTBOX } ; 
//}}AFX_DATA 

// Implementation 
protected: 

virtual void DoDataExchange(CDataExchange* pDX); // DDX/DDV 

support 

//{ { AFX_MSG(CAboutpig) 

// No message handlers 
//}}AFX_MSG 

DECLARE_MESSAGE_MAP() 



Copyright 1998 Veon, Ltd. 



wo 99/10822 



PCT/US98/17444 



286 

}; 

CAboutDlg::CAboutDlg() : CDialog(CAboutDlg::IDD) 
{ 

//{ { AFX_DATA_INIT(CAboutDlg) 
//}}AFX_DATA_INIT 

} 

void CAboutDlg::DoDataExchange(CDataExchange* pDX) 
{ 

CDialog: :DoDataExchange(pDX); 
//{ { AFX_DATA_MAP(CAboutDlg) 
//}}AFX_DATA_MAP 

} 

BEGIN_MESSAGE_MAP(CAboutDlg, CDialog) 

//{ { AFX_MSG_MAP(CAboulDlg) 
// No message handlers 

//}}AFX_MSG_MAP 
END_MESSAGE_MAP() 

// App command to run the dialog 
void CHvstreamApp::OnAppAbout() 

{ 

CAboutDlg aboutDlg; 
aboutDlg.DoModalQ; 

} 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II CHvstreamApp commands 
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// hvstream.h : main header file for the HVSTREAM application 
// 

#ifndef_AFXWIN_H_ 

#error include 'stdafx.h' before including this file for PCH 

#endif 

^include "resource.h" // main symbols 

iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiim 

II CHvstreamApp: 

// See hvstream.cpp for the implementation of this class 

// 

class CHvstreamApp : public CWinApp 
{ 

public: 

CHvstreamAppO; 
// Overrides 

// ClassWizard generated virtual function overrides 

//{ {AFX_VIRTUAL(CHvstreamApp) 

public: 

virtual BOOL InitlnstanceQ; 
//}}AFX^VIRTUAL 

// Implementation 

//{ { AFX^MSG(CHvstreamApp) 
afx_msg void OnAppAboutQ; 

// NOTE - the ClassWizard will add and remove member functions 

here. 

// DO NOT EDIT what you see in these blocks of generated 

code ! 

//}}AFX_MSG 
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DECLARE_MESSAGE_MAPO 

}; 
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llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll^ 

II hvstrvw.cpp : implementation of the CHvstreamView class 

// 

#include "stdafx.h" 
#define _OLEAUTO_H_ 
#include <vfw.h> 

#include "hvstream.h" 

#include "hvstrdoch" 
#include "hvstrvw.h" 

#define EMPTY_CODE "0" 
#define ELLIPSE_CODE " 1 " 
#defme RECT_CODE "2" 
#ifdef_DEBUG 
#undefTHIS_FILE 

static char BASED_CODE THIS_FILE[] = _FILE_; 
#endif 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II CHvstreamView 

IMPLEMENT_DYNCREATE(CHvstreamView, CView) 

BEGIN_MESSAGE_MAP(CHvstreamView, CView) 
//{ { AFX_MSG_MAP(CHvstreamView) 

ON_COMMAND(ID_STREAM_ADDSTREAM, OnStreamAddstream) 
ON_COMMAND(ID_FILE_OPEN,OnFileOpen) 
//}}AFX_MSG_MAP 
END_MESSAGE_MAP() 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
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// CHvstreamView construction/destruction 

CH vstream View: :CH vstreamViewO 

{ 

// TODO: add construction code here 

} 

CHvstreamView: :~CH vstreamViewQ 

{ 
} 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II CHvstreamView drawing 

void CHvstreamView::OnDraw(CDC* pDC) 
{ 

CHvstreamDoc* pDoc = GelDocument(); 
ASSERT_VALID(pDoc); 

// TODO: add draw code for native data here 

) 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II CHvstreamView diagnostics 

#ifdef_DEBUG 

void CHvstreamView: :AssertValid() const 
{ 

CView::AssertValid(); 

) 

void CHvstreamView: :Dump(CDumpContext«fc do) const 
{ 
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CView: :Dump(dc); 

} 

CHvstreamDoc* CHvstreamView::GelDocumenlO // non-debug version is inline 
{ 

ASSERT(m_pDocument->IsKindOf(RUNTIME_CLASS(CHvstreamDoc))); 
return (CHvstreamDoc*)inj3Document; 

} 

#endif//_DEBUG 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

II CHvstreamView message handlers 

/* 

void CHvstreamView: :OnStreamAddstream() 
{ 

int i; 

AVISTREAMINFO strhdr, videohdr; 
PAVIFILEpfile = NULL; 

PAVISTREAM psText = NULL, psVideo = NULL; 

char szText[256]; 

int iLen; 

HRESULThr; 

DWORD dwTextFormat; 

char *szTitle; // name of the avi file 

char szMessage[256]; 

DWORD numFrames; 

// get the file name 

char szFilter[] = "Video Files (*.avi) | *.avi | All Files (*.*) | *.* H"; 
CFileDialog dig (TRUE, NULL, NULL, 0, szFiiter, NULL); 
if (dlg.DoModal() !=IDOK) 
return ; 
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CString fileName = dlg.GetPathNameQ; 
szTille = strdup (dlg.m__ofn.lpstrFile); 



AVIFilelnitQ; 
// 

// Open the movie file for writing.... 

II 

hr= AVIFileOpen(&pfile, 

(LPCSTR)szTitle, 

OF^READWRITE, 

NULL); 

if(hr !=AVIERR_OK) 
goto error; 



// returned file pointer 

// file name 
// mode to open file with 
// use handler determined 
// from file extension..,. 



// Get The video stream 

hr = AVIFileGetStream (pfile, &psVideo, streamtypeVIDEO, 0); 
if(hr!=AVIERR_OK) 
goto error; 

// get the stream info 

hr = AVIStreamlnfo (psVideo, &videohdr, sizeof(videohdr)); 

// Fill in the stream header for the text stream. 

memsetC&strhdr, 0, sizeof(strhdr)); 

strhdr.fccType = streamtypeTEXT; 

strhdr.fccHandler = mmioFOURCCCD', 'R', 'A', 'W'); 

strhdr.dwScale = videohdr.dwScale; 

strhdr.dwRate = videohdr.dwRate; 

strhdr.dwSuggestedBuCferSize = sizeof(szText); 

strhdr.rcFrame = videohdr.rcFrame; 
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// SetRect (&strhdr.rcFrame, videohdr.rcFrame.left, 
videohdr.rcFrame.bottom, 

// videohdr.rcFrame.right, 
videohdr.rcFrame.bottom+30); 

SetRectEmpty(&strhdr.rcFrame); 

// ....and create the stream. 

hr = AVIFileCreateStream(pfile, &psText, (struct 
_AVISTREAMINFOW*)&strhdr); 

if(hr!=AVIERR_OK){ 

goto error; 

} 

dwTextFormat = sizeof(dwTextFormat); 

hr = AVIStreamSetForniat(psText, 0, &dwTextFormat, 
sizeof(dwTextFomiat)); 

if(hr!=AVIERR^OK) { 

goto error; 

} 

numFrames = videohdr.dwLength; 

for (i = 0; i < numFrames; i++) { 

iLen = wsprintf(szText, "%d %d %d %d %d", 0, 0, 100, 100, i); 

// DWORD dwFlags = 5 0) ? AVIIF_KEYFRAME : 0; 

// DWORD dwFlags = AVIIF_^KEYFRAME; 

DWORD dwFlags = (AVIStreamIsKeyFrame(psVideo, i)) ? 
AVIIF^KEYFRAME : 0; 

. // ... and write it as well. 

hr = AVIStreamWrite(psText, 

i, 
1, 

szText, 
iLen+ 1, 
dwFlags, 
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NULL, 
NULL); 
if(hr!=AVIERR_OK) 
break; 

CDC *pDC = GetDCO; 

wsprintf (szMessage, "frame # %d", i); 

pDC->DrawText (szMessage, -1, CRect (0, 0, 100, 100), DT_TOP | 
DT_LEFT ); 

ReleaseDC (pDC); 
} 

error: 

// 

// Now close the file 
// 

if (ps Video) 

AVIStreamClose (psVideo); 

if (psText) 

AVIStreamClose(psText); 

if(pfile) 

AVIFileClose(pfile); 

AVIFileExitO; 

} 

*/ 

void CHvstreamView: lOnStreamAddstreamQ 

{ 

inti; 

AVISTREAMINFO strhdr, videohdr; 
PAVIFILEpfile = NULL; 
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PAVISTREAM psText = NULL, psVideo = NULL; 

int iLen; 

HRESULThr; 

DWORD dwTextFormat; 

char szMessage[256]; 

DWORD numFrames; 

char *szText; 

AVIFilelnitO; 



// iterate through all the clips in the project 

for (int clipNumber = 0; clipNumber < numClips; clipNumber++) 

{ 

pfile = NULL; 

psText = psVideo = NULL; 

char fiIeNanie[256]; 

strcpy (fileName, m_diskName); 

strcat (fileName, theClips[clipNumber]->niediaFile()); 

// Open the movie file for reading and writing 

hr = AVIFileOpen(&pfile, // returned file pointer 

(LPCSTR)fileName, // file name 

OF_READWRITE, // mode to open file with 

NULL); // use handler determined 

// from file extension.... 

if(hr != AVIERR^OK) 
goto error; 

// Get The video stream 

hr = AVIFileGetStream (pfile, &psVideo, streamtypeVIDEO, 0); 
if(hr !=AVIERR_OK) 
goto error; 

// get the stream info for the video 

hr = AVIStreamlnfo (psVideo, &videohdr, sizeof(videohdr)); 
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// Fill in the stream header for the text stream. 

memset(&strhcir, 0, sizeof(strhdr)); 

strhdr.fccType = streamtypeTEXT; 

strhdr.fccHandler - mmioFOURCCCD\ •R^ *A*, 'W^; 

strhdr.dwScale = videohdr.dwScale; // 1 ; 

strhdr.dwRate = videohdr.dwRate;// / 

videohdr.dwScale; 

strhdr.dwSuggestedBufferSize = 256; //sizeof(szText); 

SetRectEmpty(&strhdr.rcFrame); 



// create the text stream. 

hr = AVIFileCreateStream(pfile, &psText, (struct 
_AVlSTREAMINFOW*)&strhdr); 

if(hr != AVIERR_OK){ 

goto error; 

} 

dwTextFormat = sizeof(dwTextFormat); 

hr = AVIStreamSetFormat(psText, 0, &dwTextFormat, 
sizeof(dwTextFormat)); 

if(hr !-AVIERR_OK) { 

goto error; 

} 

numFrames = videohdr.dwLength; 

// loop the frames to write stream data 
for (i = 0; i < numFrames; i++) 
{ 

HvClipStruct * theClip = theClips[clipNumber]; 
HvClipROI* Rois = theClip->GetRoiOfFrame(i) ; 
CString text; 
if(Rois = 0) 

text = EMPTY_CODE; 

else 
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text = ELLIPSE_CODE; 
HvObjectStruct *object; 
while (Rois != 0) 
{ 

CString roiText; 

object = GetObjectFromTrack (theClip->name(), 

theClip->GetSequence(Rois->id)); 

roiText.Format (" %d %d %d %d", Rois->x, 

Rois->y, Rois->a, Rois->b); 

roiText = roiText + " " + CString (object->id); 
text +=roiText; 
Rois = Rois->next ; 

} 

szText = strdup ((LPCTSTR)text); 
iLen = strlen (szText); 



DWORD dwFlags = (AVIStreamlsKeyFrame(psVideo, i)) 
? AVI1F_KEYFRAME : 0; 

// write the frame data. 

hr = AVlStream Write(psText, i, 1 , szText, 

iLen, dwFlags, NULL, NULL); 

if(hr!= AVIERR_OK) 
break; 



CDC *pDC = GetDCO; 

wsprintf (szMessage, "%s # %d", theClipsPaths[clipNumber], i); 

pDC->DrawText (szMessage, -1, CRect (0, 0, 300, 100), 
DT_TOP I DT_LEFT ); 

ReleaseDC (pDC); 
} 



error: 

II 
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// Now close the file 
// 

if (psVideo) 

AVIStreamClose (psVideo); 

if (psText) 

AVIStreamClose(psText); 

if(pfile) 

AVIFileClose(pfile); 

} 

AVIFileExitQ; 

CDC *pDC = GetDCQ; 

wsprintf (szMessage, "Writing terminated"); 

pDC->DrawText (szMessage, -1, CRect (0, 0, 300, 100), DT_TOP | 
DT_LEFT); 

ReleaseDC (pDC); 

} 

void CHvstreainView::OnFileOpen() 
{ 

// get the file name 

char s2Filter[] = "Obvious Project (*.obn) | *.obn | All Files (*.*) | *.* H"; 
CFileDialog dig (TRUE, NULL, NULL, 0, szFilter, NULL); 
if(dlg.DoModalO !=IDOK) 
return ; 

CString fileName = dlg.GetPathName(); 

CString l_sCaption = "HV Stream Writer - " + fileName; 

SetWindowText (l_sCaption); 
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//allocate HvProject object 
DeleteProjectO; 

theProject = new HvProject ((char *)fileName.GetBuffer(256)); 
// set current directory to the correct machine 
m_diskName[0] = fileName[0]; 
m_diskName[l] = 
m_diskName[2] = ^0'; 
int rc; 

//retrieve the names of the clips in the file and create HvClip objects 
numCiips = theProject->GetClips(&theClipsPaths) ; 
for (int i = 0 ; i < numClips ; i++) 
{ 

// get the full clip name 

char * clipName = new char[256]; 

strcpy (clipName, m_diskName); 

strcat (clipName, theClipsPaths[i]); 

theClips[i] = new HvClipStruct (rc, strdup(clipName) ) ; 

} 

//retrieve the names of the objects in the file and create HvObject objects 

numObjects = theProject->GetObjects(&theObjectsPaths) ; 
for ( i = 0 ; i < numObjects ; i-H-) 
{ 

// get the full object name 

char * objectName = new char[256]; 

strcpy (objectName, m_diskName); 

strcat (objectName, theObjectsPaths[i]); 

theObjects[i] = new HvObjectStruct (rc, strdup(objectName) ); 

} 
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} 

void CHvstreamView::DeleteProject() 

{ 

// free all project data 

} 

HvObjectStruct* CHvstreamView::GetObjectFromTrack(char* clipName, char* 
clipSequence) 

{ 

for(inti = 0;i<20;i-H-) 
{ 

if(theObjects[i] !=0) 
{ 

if (theObjects[i]->ExistObj Sequence(clipName, 

clipSequence)) 

return theObjects[i] ; 

} 

} 

return 0 ; 

} 
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// hvstrvw.h : interface of the CHvstreamView class 
// 

llllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllim 

#include "hvclp.h" 

#include "hvobj.h" 

#include "hvprj.h" 

#include "hvflstr.h" 

class CHvstreamView : public CView 

{ 

protected: // create from serialization only 
CHvstreamViewQ; 

DECLARE_DYNCREATE(CHvstreamView) 

// protected data members 

HvProject *theProject; 

HvClipStruct* theClips[20); 
char **theClipsPaths; 
int numClips; 

HvObjectStruct* theObjects[20]; 
char **theObjectsPaths; 
int numObjects; 

char m_diskName[3]; 
// Attributes 
public: 

CHvstreamDoc* GetDocumentQ; 

// Operations 
public: 

// Overrides 
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// ClassWizard generated virtual function overrides 
//{ { AFX_VIRTUAL(CHvstreaniView) 
public: 

virtual void OnDraw(CDC* pDC); // overridden to draw this view 
protected: 

//}}AFX_VIRTUAL 

// Implementation 
public: 

virtual --CHvstreamViewO; 
#ifdef_DEBUG 

virtual void AssertValid() const; 

virtual void Dump(CDumpContext& dc) const; 

#endif 

protected: 

// free project allocated memory 
void DeleteProjectQ; 

// get an object from a given clip and sequence 

HvObiectStruct* GetObjectFromTrack(char* clipName, char* 
clipSequence); 

// Generated message map functions 
protected: 

//{ { AFX_MSG(CHvstreamView) 

afx_msg void OnStreamAddstreamQ; 

afx_msg void OnFileOpenQ; 

//}}AFX_MSG 

DECLARE^MESSAGE_MAP() 

}; 

#ifndef _DEBUG // debug version in hvstrvw.cpp 
inline CHvstreamDoc* CHvstreamView::GetDocument() 
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{ return (CHvstreamDoc*)m_pDocument; } 
#endif 



lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
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'.^ Mr*?*"" 
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// mainfrm.cpp : implementation of the CMainFrame class 
// 

^include "stdafx.h" 
^include "hvstream.h" 

#include "mainfrm.h" 

#ifdef_DEBUG 
#undefTHIS_FILE 

static char BASED_CODE TH1S_FILE[] = _FILE_; 
#endif 

iiiiiiiiiiiiitiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiniiiii 

II CMainFrame 

IMPLEMENT_DYNCREATE(CMainFrame, CFrameWnd) 

BEGrN_MESSAGE_MAP(CMainFrame, CFrameWnd) 
//{ {AFX_MSG_MAP(CMainFrame) 

// NOTE - the ClassWizard will add and remove mapping macros 

here. 

// DO NOT EDIT what you see in these blocks of generated 

code ! 

//}}AFX_MSG_MAP 
END_MESSAGE_MAP() 

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIHIIIIIIIIIIIIIII 
II CMainFrame construction/destruction 

CMainFrame: :CMainFrame() 
{ 

// TODO: add member initialization code here 

} 
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CMainFrame: i-CMainFrameQ 

{ 
} 

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 
II CMainFrame diagnostics 

#ifdef_DEBUG 

void CMainFrame::AssertValid() const 
{ 

CFrameWnd :: Assert ValidQ; 

} 

void CMainFrame: :Dump(CDumpContext& dc) const 
{ 

CFrameWnd ::Dump(dc); 

} 

#endif//_DEBUG 
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llllllllllllllllllllllllllllllllllllllllllllllllllllllllim^^^ 
II CMainFrame message handlers 
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// mainfrm.h : interface of the CMainFrame class 
// 

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIU^ 

class CMainFrame : public CFrameWnd 

{ 

protected: // create from serialization only 
CMainFrameQ; 

DECLARE_DYNCREATE(CMainFrame) 

// Attributes 
public: 

// Operations 
public: 

// Overrides 

// ClassWizard generated virtual function overrides 
//{ { AFX^VIRTUAL(CMainFrame) 
//}}AFX^VIRTUAL 

// Implementation 
public: 

virtual -CMainFrame(); 
#ifdef_DEBUG 

virtual void AssertValidQ const; 

virtual void Dump(CDumpContext& dc) const; 

#endif 

// Generated message map functions 
protected: 

//{ { AFX_MSG(CMainFrame) 

// NOTE - the ClassWizard will add and remove member functions 

here. 
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// DO NOT EDIT what you see in these blocks of generated 

code! 

//}}AFX_MSG 

DECLARE_MESSAGE_MAPO 

); 
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lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

II { {NO_DEPENDENCIES} } 

// Microsoft Visual C++ generated include file. 

// Used by hvstream.rc 

// 

^define IDD_ABOUTBOX 100 
#define IDR_MAINFRAME 128 
#define ID_STREAM_ADDSTREAM 32771 

// Next default values for new objects 

// 

#ifdef APSTUDIO_IN YOKED 

#ifndef APSTUD10_READ0NLY_SYMB0LS 

#define _APS_3D_C0NTR0LS 1 

#define _APS_NEXT_RESOURCE_VALUE 130 

#define _APS_NEXT_COMMAND_ VALUE 32772 

#define _APS_NEXT_CONTROL_VALUE 1 000 

#define _APS_NEXT_S YMED_V ALUE 1 0 1 

#endif 

#endif 
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// stdafxxpp : source file that includes just the standard includes 



// 



hvstream.pch will be the pre-compiled header 



// 



stdafx.obj will contain the pre-compiled type information 



#include "stdafx.h" 

/* HV Project 

Ephyx Technologies Ltd, 

Copyright © 1994. All Rights Reserved. 

SUBSYSTEM: hvtest.exe Application 
FILE: hvclp.cpp 



OVERVIEW 



Class member methods for a clip file. 

file structure: 
<clip name> 
<video file full path> 

<video type> // dropped for the time being 

<num markers> each marker is <frame number> , <identifier> 



<num sequences> 

<sequence> each sequence is <seq number> , <identifier> 



<markerl> 
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frame describe the relevant ROIs. 

<t> <id> <x> <y> <a> <b> <theta> 



#include "stdafx.h" 
#include "hvclp.h" 
#include "hvflstr.h" 

#include "iostream.h" 
#include "fstream.h" 

static char flstr[255]; 

// 

// definition ctor 

// 

HvClipStruct::HvClipStruct(int &rc, const char *name, HvVideoTypes type) : 
_type(type) 

{ 

rc = HV_OK ; 

maxFramesForRoi = _numMarkers = 0 ; 

// start with an allocation of markers 
_numAllocatedMarkers = 4 ; 

_listOfMarkers = 0 ; 
JistOfRoiByFrame = 0 ; 

JistOfMarkers = new HvClipMarker |_numAllocatedMarkers] ; 
if(_listOfMarkers = 0) 
{ 

rc = HV_ERROR; 



ROIS: for each 
( where id is of the sequence ) 



Copyright 1998 Veon, Ltd. 



wo 99/10822 



PCT/US98/17444 



313 

return ; 

} 

numAllocatedSequences = 4 ; 
_numSequences = 0 ; 

bagOfSequences = new HvSequence [ numAllocatedSequences ] ; 
if (_bagOfSequences = 0) 

{ 

rc = HV_ERROR; 
return ; 

} 

name = 0 ; 

// copy the given name 
if (name != 0) 

{ 

name = new char [strlen(name)+l] ; 
if (_name != 0 ) 
strcpy(_name, name) ; 

) 

_mediaFilePath = 0 ; 

} 
// 

// a ctor from a file path: 

// opens the file and reads the info 

// rc = -l if fails 

HvClipStruct::HvClipStruct( int &rc, const char *filePath ) : _type(HV_AVl) 
{ 

FILE *fileDesc ; 
char line[256] ; 
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rc = HV_OK ; 

if (filePath != 0 && filePath[0] != '\0') 
{ 

// open the file for read only 
if ((fileDesc = fopen(filePath, "rb")) = 0 ) 
{ 

rc = HV_ERROR ; 
return ; 

} 

// get the name of the clip 

if (!EGetSlr(fileDesc,flstr,255)) 

{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

sscanf(flstr,"%s",line) ; 

// allocate and copy the data 

if ((_name = new char[strlen(line) + 1]) == 0) 

{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

strcpy(_name, line) ; 

// get the full path of the media file 
if(!EGetStr(fileDesc,flstr,255)) 

{ 

rc = HV_ERROR ; 
fclose(fiieDesc) ; 
return ; 
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} 

sscanf(flstr,"%s",line) ; 
// allocate and copy the data 

if ((_mediaFilePath = new char[strlen(line) + 1]) = 0) 
{ 

rc = HV_ERROR; 
fclose(fileDesc) ; 
return ; 

} 

strcpy(_mediaFilePath, line) ; 

// get the video type 

// the type is assumed AVI 

Jype = HV_AVI ; 

// read the markers set for the clip 
if(!EGetStr(fileDesc,flstr,255)) 

{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

sscanf(flstr,"%d",&_numMarkers); 

// allocate and copy the data ( allocate a bit more in case we'll 
need to add later 

if (_numMarkers > 0) 
{ 

JistOfMarkers = new HvClipMarker 

|_numMarkers+3] ; 

_numAllocatedMarkers = _numMarkers + 3 ; 

ifOistOfMarkers = 0) 
{ 

rc = HV ERROR ; 
fclose(fileDesc) ; 
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return ; 

} 

} 

else 

{ 

numAIIocatedMarkers = 0 ; 
JistOfMarkers = 0 ; 

} 



line) ; 



// loop and read all the markers 
for ( int i = 0 ; i < _numMarkers ; i++) 
{ 

if(!EGetStr(fileDesc,flslr,255)) 
{ 

rc = HV_^ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

sscanf(flstr, "%ld %s", &_listOfMarkers[i].mark, 
_listOfMarkers[i].id = strdup(line) ; 

} 

// read the sequences for the clip 
if (!EGetStr(fileDesc,flstr,255)) 

{ 

rc = HV_ERROR; 
fclose(fileDesc) ; 
return ; 

} 

sscanf(flstr,"%d",&_numSequences) ; 

// allocate and copy the data ( allocate a bit more in case 



we'll need to add later 

if (_numSequences > 0) 

{ 
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bagOfSequences = new HvSequence 

LnumSequences +3] ; 

_numAIlocatedSequences = _numSequences +3 ; 

if (_bagOfSequences = 0) 
{ 

rc - HV^ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

) 

else 

{ 

_numAllocatedSequences = 0 ; 
_bagOfSequences = 0 ; 

} 



line) 



// loop and read all the sequences 
for ( i = 0 ; i < numSequences ; 1++) 

{ 

if (!EGetStr(fileDesc,flstr,255)) 
{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

sscanf(flstr, "%ld %s", &_bagOfSequences[i].id, 
_bagOfSequences[i].name = strdup(line) ; 



// loop and read all ROIs 
maxFramesForRoi = 0 ; 

{ 

int t,a,b,x,y; 



Copyright 1998 Veon, Ltd. 



■WO 99/1 0822 



PCT/US98/I7444 



318 



long hnd; 
float th ; 



%f •,&t,&hnd,&x,&y,&a,&b,&th) ; 



slots 



while (EGetStr(fileDesc,flstr,255)) 
{ 

sscanf(flstr,"%d %ld %d %d %d %d 



HvClipROI(hnd, x, y, a, b, th. 



if (t >= maxFramesForRoi) 

// allocate another chunk of frame 

if(!anocFrameSlots(t)) 
{ 

rc = HV_ERROR ; 
fclose(fileDesc) ; 
return ; 

} 

// put the data in the right slot 
if (_listOflloiByFrame[t] != 0) 
{ 

HvClipROI* temp = new 



_listOfiloiByFrame[t],0 ) ; 



temp ; 



HvClipROI(hnd, x, y, a, b, th, 



} 

else 
{ 



_listOfRoiByFrame[t]->prev = 
listOfRoiByFrame[t] = temp ; 



JistOfRoiByFrame[t] = new 



.listOfRoiByFrame[t],0 ) ; 



} 



// connect it to the sequence list 
Copyright 1998 Veon, Ltd. 



-WO 99/10822 



PCT/US98/I7444 



319 

AddToSequence(_listOfRoiByFrame[t]) ; 

} 

) 

fclose(fileDesc) ; 

} 

else 

{ 

rc = HV_ERROR ; 

} 

return ; 

} 

// 

// a virtual dtor 

// 

HvClipStructii-HvClipStructQ 
{ 

HvClipROI* temp ; 

if (_name != 0) 
delete _name ; 

if(_mediaFilePath !=0) 
delete _mediaFilePath ; 

if (_numMarkers > 0) 
delete listOfMarkers ; 

if (maxFramesForRoi > 0) 
for (int i = 0 ; i < maxFramesForRoi ; i++) 

if (( temp = JistOfRoiByFrame[i]) != 0 ) 
while (temp) 
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{ 

HvClipROI *cur = temp ; 
temp = temp->next ; 
delete cur ; 

} 

} 

void HvClipStruct::selName(const char *name) 
{ 

if (name = 0) 
return ; 

if (_name) 

delete _name ; 

_name = new char [ strlen(name) + 1 ] ; 
if (_name != 0 ) 

strcpy(_name , name ) ; 

return ; 

} 

void HvClipStruct::setMediaFile(const char *name) 
{ 

if (name == 0) 
return ; 

if(_mediaFilePath) 

delete _mediaFilePath ; 

mediaPilePath = new char [ strlen(name) + 1 ] ; 
if(_mediaFilePath !=0) 

strcpy(_mediaFilePath , name ) ; 
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return ; 

} 

// 

// AddToSequence: adds a ROl to a sequences (always at the end) 
// 

void HvClipStruct::AddToSequence(HvClipROI *roi) 
{ 

int set = -1 ; 

// find the right sequence 

for (int i = 0 ; i < numSequences ; i++) 

if (_bagOfSequences[i].id = roi->id) 
set = i ; 

if(set<0) 

return ; // no such sequence defined (must define a sequence 

first ) 

// assume it is a new LAST roi 

if (_bagOfSequences[set]. first == 0) // no rois until now 

_bagOfSequences[set]. first = _bagOfSequences[set] Jast = roi ; 

else 

{ 

_bagOfSequences[set].last->nextInSequence = roi ; 
_bagOfSequences[set].last = roi ; 

} 

} 

// 

// GetSequence : retrieves a sequence name 

// 

// 

char* HvClipStruct::GetSequence( long id ) 
{ 
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if(id<0) 
return 0 ; 

for (int i = 0 ; i < _nuinAllocatedSequences ; i-H-) 
{ 

if (_bagOfSequences[i].id == id) 

{ 

return _bagOfSequences[i].name ; 

} 

} 

return 0 ; 

) 

II 

II DelSequence : deletes a sequence to the clip's list 
// 

// return HV_OK if all OK, otherwise HV_ERROR 

// 
// 

int HvClipStruct::DelSequence( const char* name ) 
{ 

if (name == 0) 
return HV_ERROR ; 

for (int i = 0 ; i < _numAllocatedSequences ; i++) 
{ 

if (strcmp(_bagOfSequences[i].name5name) == 0) 
return DelSequence(_bagOfSequences[i].id) ; 

} 

return HV_ERROR ; 

} 

// 
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// DelSequence : deletes a sequence to the clip's list 
// 

// return HV_OK if all OK, otherwise HV_ERROR 

// 

int HvClipStruct::DelSequence( long id ) 
{ 

if (id < 0) 

return HV_ERROR ; 



for (int i = 0 ; i < _nuniAllocatedSequences ; i++) 

{ 

if (_bagOfSequences[i].id == id) 
{ 

// loop to free the ROIs 

HvClipROI* current = _bagOfSequences[i]. first ; 
HvClipROI* temp ; 
while (current != 0) 

{ 

if (current->prev != 0) // BUG: we do not 



how to clear first elements !!! 



current->next ; 



{ 



current->prev ; 



current->prev->next = 

if (current->next != 0) 

current->next->prev 



} 

else 
{ 



temp = current ; 

current = current->nextlnSequence 
delete temp ; 

// all we do is skip 

current = current->nextInSequence 
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} 

) 

_bagOfSequences[i].id = OL ; 
_numSequences~ ; 

if (_numSequences < 0) _numSequences = 0 ; 
return HV_OK ; 



} 

} 

return HV_ERROR ; 



} 



long HvClipStruct::GetSequenceId() 
{ 

// start with numsequences+1 and make sure 
long id = _numSequences+l ; 

while (IdExists(id)) id++ ; 

return id ; 

) 

int HvClipStruct::IdExists(const int id) 
{ 

for (int i = 0; i < _numAllocatedSequences; i+-i-) 
if (_bagOfSequences[i].id == id) 
return 1 ; 

return 0 ; 

} 
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// 

// AddClipSequence : adds a sequence to the clip's list 
// 

// return HV_OK if all OK, otherwise HV_ERROR 

// 

int HvClipStruct::AddClipSequence( const char *name ) 

{ 

if (name == 0) 
return HV^ERROR ; 

if (_numSequences >= _numAllocatedSequences) 
// allocate another chunk of frame slots 
if (lallocSequenceSlotsQ) 
return HV^ERROR ; 

long id = GetSequenceldQ ; 

// find an empty spot 
int set = 0; 

while (_bagOfSequences[set].id != 0) set-H- ; 

// put the data in the right slot 
_bagOfSequences[set].id = id ; 
_bagOfSequences[set].name = strdup(name) ; 

_numSequences-H- ; 
return id ; 

} 

// 

// GetSequences : returns the number and names of all sequences defmed 

// 

int HvClipStruct::GetSequences(char *** names) 
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{ 

int total = 0; 

if (_numSequences == 0) 
return 0 ; 

(*names) = new char* LnumSequences] ; 
if ((*names) == 0) 
return HV^ERROR ; 

for (int i == 0 ; i < _numAllocatedSequences; 1++) 
{ 

if (_bagOfSequences[i].id != OL) // then it is a sequence 

(* names) [total+-f-] = strdup(_bagOfSequences[i].name) ; 

} 

return total ; 

) 

// 

// GetSequencesExt : returns the number and names of all sequences defined 

// concatenating the clip's 

name at the beginning of each sequence. 

// 

int HvClipStruct::GetSequencesExt(char *** names) 
{ 

int total = 0; 

if (_numSequences == 0) 
return 0 ; 

(♦names) = new char* |_numSequences] ; 
if((*names) = 0) 
return HV ERROR; 
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for (int i = 0 ; i < numAllocatedSequences; 
{ 

if (_bagOfSequences[i].id != OL) // then it is a sequence 
{ 

(*names)[total] = new 
char[strlen(_name)+strlen(_bagOfSequences[i] .name)+3] ; 

sprintf((*names)[totaH-+],"%s::%s",_name, 
_bagOfSequences[i].name) ; 

} 

} 

return total ; 

} 



// 

// addMarker : adds a marker to the clip's list 
// 

// return HV_OK if all OK, otherwise H V_ERROR 
// 

int HvClipStruct::addMarker( long t,char *id ) 
{ 

if(t<0|| id==0) 
return HV_ERROR ; 

if (_numMarkers >= _numAllocatedMarkers) 
// allocate another chunk of frame slots 
if(!allocMarkerSlots()) 
return HV_ERROR ; 

// put the data in the right slot 
JistOfMarkersLnumMarkers].mark = t ; 
JistOfMarkersLnumMarkersj.id = strdup(id) ; 
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_numMarkers++ ; 
return HV_OK ; 

} 

// 

// addRoi : adds a ROI element to the clip's list 
// 

// return HV_OK if all OK, otherwise HV_ERROR 
// 

int HvClipStruct::addRoi( long t,long handle, int x, int y, int a, int b, float theta ) 
{ 

if (t >= maxFramesForRoi) 

// allocate another chunk of frame slots 

if(!allocFrameSlots(t)) 

return HV_ERROR ; 

// put the data in the right slot 

if (_HstOfRoiByFrame[(int)t] == 0) 

listOfRoiByFrame[(int)t] = new HvClipR01(handle, x, y, a, b, theta, 0, 

0); " 
else 

{ 

HvClipROI* temp = new HvClipROI(handle, x, y, a, b, theta, 

_listOfRoiByFrame[(int)t],0 ) ; 
_listOfRoiByFrame[(int)t]->prev = temp ; 
_listOfRoiByFrame[(int)t] = temp ; 

} 

// connect it to the sequence list 
AddToSequence(_listOfRoiByFrame[t]) ; 

return HV_OK ; 

} 
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long HvClipStruct::ObjectAtFramePoint(const long t, const int x, const int y) 
const 

{ 

long hnd ; 

if(t<0) 
return -IL ; 

if (t >= maxFramesForRoi) 
return -IL ; 

HvClipROI* temp = JistOfRoiByFrame[(int)t] ; 

// go thru the list of frames at time t 
while (temp != 0) 
{ 

if (( hnd = temp->PtInROI(x,y) ) != -IL) 

return hnd ; 
temp = temp->next ; 

} 

// did not find anything 
return -IL ; 

} 
// 

// allocFrameSlots :: increase memory allocation for ROIs 

// 

int HvClipStruct::allocFrameSlots(const long minPrame) 

{ 

int upTo = (( (int)minFrame >= ( 2 * maxFramesForRoi )) ? 
(int)minFrame +1:2* maxFramesForRoi ) ; 

//alloc at least 10 

upTo = ( upTo < 10 ) ? 10 : upTo ; 



Copyright 1998 Veon, Ltd. 



wo 99/10822 



PCT/US98/17444 



330 

// alloc a new array 

HvClipROI** temp = new HvClipROI* [upTo] ; 

// alloc problems 
if(temp = 0) 
return 0 ; 

// clear the array 
for (int i = 0 ; i < upTo ; i-H-) 
temp[i] = 0 ; 

if (_HstOfRoiByFrame != 0) 
{ 

// copy what we already have 
for (i = 0 ; i < maxFramesForRoi ; i++) 
temp[i] = JistOfRoiByFrame[i] ; 

) 

// set the new max number 
maxFramesForRoi = upTo ; 

JistOfRoiByFrame = temp ; 

return 1 ; 

} 

// 

// allocMarkerSlots :: increase memory allocation for markers 

// 

int HvClipStruct: lallocMarkerSlotsQ 
{ 

HvClipMarker* temp = new HvClipMarkerLnumAUocatedMarkers + 10 

] ' 
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if(temp = 0) 
return 0 ; 

numAllocatedMarkers += 10 ; 

// copy the old markers 
for (int i = 0 ; i < _numMarkers ; i-H-) 
temp[i] = JistOfMarkers[i] ; 

delete listOfMarkers ; 

listOfMarkers = temp ; 

return 1 ; 

} 

// 

// allocSequenceSlots :: increase memory allocation for sequences 

// 

int HvClipStruct::allocSequenceSlots() 
{ 

HvSequence* temp = new HvSequenceLnumAllocatedSequences + 10 ] ; 

if (temp = 0) 
return 0 ; 

numAllocatedSequences += 10 ; 

// copy the old markers 
for (int i = 0 ; i < _numSequences ; i++) 
temp[i] = _bagOfSequences[i] ; 

delete _bagOfSequences ; 
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_bagOfSequences = temp ; 
return 1 ; 

} 

// 

// a method to write a clip file 

// 

int HvClipStruct::Write(const char *filePath) 
{ 

ofstream outfile ; 

if(filePath = 0) 
return HV_ERROR ; 

outfile.open(filePath, ios::out ) ; 

Write(outfile) ; 

outfile.closeO ; 

return HV_OK ; 

} 

// 

// a debug method to dump to file 

// 

int HvClipStruct::Dump(const char *filePath) 
{ 

ofstream outfile ; 

if(filePath = 0) 
return HV_ERROR ; 
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outfile.open(fiIePath, ios::out ) ; 
Dump(outfile) ; 
return HV_OK ; 

} 

// 

// a mathod to write a clip file 

// 

void HvClipStruct::Write(ostreain& os) 
{ 

OS « _name « "\n" ; 
OS « _mediaFilePath « "\n" ; 
// OS « " video type: AVI \n" ; 

OS « _numMarkers « "\n" ; 
for (int i =0 ; i< _numMarkers ; 

OS « _listOfMarkers[i].mark « " " « _listOfMarkers[i].id « "\n" ; 

OS « _numSequences « "\n" ; 
for (i =0 ; i< _numAllocatedSequences ; \++) 
if (_bagOfSequences[i].id != 0) 
OS « _bagOfSequences[i].id « " " «_bagOfSequences[i].name « 

"\n" ; 

for (i = 0 ; i < maxFramesForRoi ; 
{ 

HvClipROI* temp = _listOfRoiByFrame[i] ; 
if (temp !=0) 

while (temp ) 
{ 

OS « i « " " « temp->id « " " ; 

OS « temp->x « " " « temp->y « " " ; 
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OS « temp->a « " " « temp->b « " " ; 
OS « temp->theta « "\n" ; 
temp = temp->next ; 

} 



// a debug mathod to dump all info 

// 

void HvClipStruct::Dump(ostream& os) 
{ 

int i; 

OS « " dumping an HvCIipStruct" « "\n" ; 

OS « " clip name: " « _name « "\n" ; 

OS « " video file path: " « _mediaFilePath « "\n" ; 

OS « " video type: AVI \n" ; 

OS « " markers defined: " « _numMarkers « "\n" ; 

for ( i = 0 ; i< _numMarkers ; i++) 

OS « " \"" « _listOfMarkers[i].id « "\" at " « 
JistOfMarkers[i].mark « "\n" ; 

OS « " sequences defined: " « _numSequences « "\n" ; 
for ( i = 0 ; i< _numAllocatedSequences ; 
if (_bagOfSequences[i].id != 0) 

OS « " \"" «_bagOfSequences[i].name « " referenced: " « 
_bagOfSequences[i].id « "\n" ; 

OS « " ROIs defined: \n" ; 

for (i = 0 ; i < maxFramesForRoi ; i++) 

{ 

HvCiipROI* temp = JistOfRoiByFrame[i] ; 
if (temp !=0) 
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temp->y « "\n" ; 
temp->b « 



while (temp ) 
{ 



OS « " object: " « temp->id « "\n" ; 
os«" in frame "«i«"\n"; 
os«" placed at " « temp->x « « 

OS « " parameters " « temp->a « "," « 

"," « temp->theta « "\n" ; 
temp = temp->next ; 



} 

OS « " end of dump" « "\n" ; 



} 



// 

// Class HvSequence 

// 

HvSequence: :HvSequence(long id, char *name,HvClipROI* first, HvClipROI* 
last) 

{ 

this->id = id ; 
if (name !=0) 

this->name = strdup(name); 
this->first = first ; 
this->last = last ; 

} 

HvSequencelterator: :HvSequenceIterator(HvSequence* sq) 
{ 

seq = sq ; 
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current = sq->first ; 

} 

HvClipROI* HvSequenceIterator::Next() 
{ 

if (current !=0) 
{ 

current = current->nextInSequence ; 

} 

return current ; 

} 

HvClipROI* HvSequenceIterator::First() 
{ 

if (current != 0) 
{ 

if(seq !=0) 

current = seq->first ; 

} 

return current ; 

} 

HvClipROI* HvSequencelterator::Last() 
{ 

if (current != 0) 
{ 

if(seq !=0) 

current = seq->last ; 

} 

return current ; 

} 

long HvClipROI ::PtInR01(const int x , const int y ) const 
{ 
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float xl = (float)(x - this->x) ; 
float yl = (float)(y - this->y) ; 
if ( ((xl / a) * (xl / a) + (yl / b) * (yl / b)) > 1.0) 
return -IL ; 

return id ; 

} 



MICROSOFT FOUNDATION CLASS LIBRARY : HVSTREAM 



App Wizard has created this HVSTREAM application for you. This application 
not only demonstrates the basics of using the Microsoft Foundation classes 
but is also a starting point for writing your application. 

This file contains a summary of what you will find in each of the files that 
make up your HVSTREAM application. 
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HVSTREAM.MAK 

This project file is compatible with the Visual C++ development 
environment. 

It is also compatible with the NMAKE program provided with Visual C++. 

To build a debug version of the program from the MS-DOS prompt, type 
nmake /f HVSTREAM.MAK CFG=" Win32 Debug" 

or to build a release version of the program, type 
nmake /f HVSTREAM.MAK CFG="Win32 Release" 

HVSTREAM.H 
This is the main header file for the application. It includes other 
project specific headers (including RESOURCE.H) and declares the 
CHvstreamApp application class. 

HVSTREAM.CPP 
This is the main application source file that contains the application 
class CHvstreamApp. 

HVSTREAM.RC 

This is a listing of all of the Microsoft Windows resources that the 
program uses. It includes the icons, bitmaps, and cursors that are stored 
in the RES subdirectory. This file can be directly edited in the 
Visual C++ development environment. 

o 

RES\HVSTREAM.ICO 

This is an icon file, which is used as the application's icon. This 
icon is included by the main resource file HVSTREAM.RC. 

RES\HVSTREAM.RC2 

This file contains resources that are not edited by the Visual C++ 
development environment. You should place all resources not 
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editable by the resource editor in this file. 

HVSTREAM.CLW 
This file contains information used by Class Wizard to edit existing 
classes or add new classes. Class Wizard also uses this file to store 
information needed to create and edit message maps and dialog data 
maps and to create prototype member functions. 

Illllllllllllllllllllllllllllllllllllllllllllllin^ 

For the main frame window: 

mainfrm.H, mainfrm.CPP 

These files contain the frame class CMainFrame, which is derived from 
CFrameWnd and controls all SDI frame features. 

Illllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll 

AppWizard creates one document type and one view: 

hvstrdoc.H, hvstrdoc.CPP - the document 

These files contain your CHvstreamDoc class. Edit these files to 

add your special document data and to implement file saving and loading 

(via CHvstreamDoc:: Serialize). 

hvstrvw.H, hvstrvw.CPP - the view of the document 
These files contain your CHvstreamView class. 
CHvstreamView objects are used to view CHvstreamDoc objects. 
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iiiiiiimiimiiuiiiiiiiiiiiiiiiiiiiu^ 

Other standard files: 

STDAFX.H, STDAFX.CPP 
These files are used to build a precompiled header (PCH) file 
named HVSTREAM.PCH and a precompiled types file named STDAFX.OBJ. 

RESOURCE.H 

This is the standard header file, which defines new resource IDs. 
Visual C+-f reads and updates this file. 

Illlllllllllllllllllllllllllllllllllllllllllllllllllllllllllll^ 
Other notes: 

App Wizard uses "TODO:" to indicate parts of the source code you 
should add to or customize. 

Illlllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll^ 
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We claim: 

1 . A data stream, comprising a hypervideo data stream including hypervideo 
data associated with an instance in time in a video, 

2. The data stream of claim 1 , wherein the hypervideo data stream further 
5 comprises target data associated with the instance in time. 

3. The data stream of claim 1 , wherein the hypervideo data stream further 
comprises geometry data of a hotspot associated with the instance in time. 

4. The data stream of claim 3, wherein the hypervideo data stream further 
comprises shape data of a hotspot associated with the instance of time. 

10 5. The data stream of claim 1 , wherein the instance in time corresponds to a 
frame of the video. 

6. A multimedia stream, comprising: 

a hypervideo data stream including, 

a header, including static hypervideo data; 

1 5 a body, coupled to the header, including dynamic hypervideo data; 

and 

wherein the dynamic hypervideo data includes time data that is 
associated with an instance of time in a video with which the dynamic hypervideo 
data and the static hypervideo data are associated. 

20 7. The multimedia stream of claim 6, wherein the static hypervideo data 
comprises target data. 

8. The multimedia stream of claim 6, wherein the dynamic hypervideo data 
further comprises geometry data of a hotspot. 
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9. The multimedia stream of claim 6, wherein the dynamic hypervideo data 
further comprises shape data of a hotspot. 

1 0. The multimedia stream of claim 6, further comprising a video stream 
including a video frame, wherein the instance in time corresponds to the video 

5 frame. 

1 1 . The multimedia stream of claim 6, further comprising an audio stream. 

12. A method of creating a multimedia stream, comprising: 
authoring a hypervideo, including a video; and 

exporting hypervideo parameters into a hypervideo data stream of a media 

10 file. 

1 3 . The method of claim 1 1 , further comprising exporting a data stream of the 
video into the media file. 

1 4. A method for authoring a hypervideo that is dynamic, comprising: 

authoring a hypervideo; and 

1 5 defining a hypervideo parameter that may vary during the hypervideo 

performance. 

1 5. The method of claim 14, further comprising defining an event handler that 
issues a command to vary the parameter upon receiving an event message from 
the hypervideo. 
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1 6. The method of claim 1 5, wherein the defining of the event handler 
comprises defining an event handler defined by a logical condition including a 
variable defined by a query to a database. 

1 7. The method of claim 1 4, wherein authoring the hypervideo comprises 
5 defining a hotspot and linking the hotspot to a target. 

1 8. The method of performing a hypervideo that is dynamic, comprising: 
performing a hypervideo; and 

varying a parameter of the hypervideo during the performance of the 
hypervideo. 

10 19. The method of claim 1 8, wherein the varying the parameter comprises 
varying the parameter upon the occurrence of an event in the hypervideo. 

20. The method of claim 19, wherein the varying the parameter upon the 
occurrence of an event in the hypervideo comprises varying the parameter upon 
an event handler receiving a message associated with the event, and issuing a 

1 5 command to vary the parameter. 

21 . The method of claim 20, wherein the varying the parameter upon an event 
handler receiving a message associated with the event, and issuing a command to 
vary the parameter comprises varying the parameter upon the event handler 
evaluating a logical condition including a variable defined by a query to a 

20 database. 

22. The method of claim 1 8, wherein the varying the parameter comprises 
varying the parameter that identifies a target. 

23. A hypervideo system, comprising: 
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a dynamic hypervideo server; 

a player coupled to the dynamic hypervideo server; and 

a media server coupled to the player. 

24. The hypervideo system of claim 23, further comprising a database. 

5 25. The hypervideo system of claim 23, further comprising an appHcation 
program. 

26. The hypervideo system of claim 23, wherein the application program is an 
electronic commerce server. 

27. The hypervideo system of claim 23, wherein the application program is an 
10 advertising server. 

28. The hypervideo system of claim 23, wherein the application program is a 
training server. 

29. The hypervideo system of claim 23, wherein the media server is a video 
server. 

1 5 30. The hypervideo system of claim 23, further comprising a network 
coupling the player to the dynamic hypervideo server and the media server. 
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